Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving end-to-end test reliability

frantic.im

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Improving end-to-end test reliability

#11

Post by vthommeret »

I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev

It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it:

1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability

2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress.

3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box.

4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc...

5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful.

My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.

Re: Improving end-to-end test reliability

#12

Post by bluGill »

sidlls wrote: post #6
martinald wrote: post #4

Earlier quoted context omitted.

The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem. Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your o…

That's a rare attitude to have in the Bay Area anyway. Unfortunately. Everyone wants lots of unit tests because they run fast and give (roughly) instant feedback. Unit tests paint an incomplete picture. Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.

The problem I have with unit tests is they inhibit refactoring off the API they test.

Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a while the company library implementers.

For everyone else we are writing to a business requirement that isn't well understood and may change. However the purpose of a unit test is to assert that no matter what this won't change. So every time you want to make a change all those tests are in the way of the change and need to be fixed.

Everyone writing tests needs to figure out their own middle ground. Because end to end tests have their own problems.

Re: Improving end-to-end test reliability

#13

Post by domesticsimian »

8organicbits wrote: post #7

> When an E2E test is failing consistently and nobody cares to fix it, that means the test isn’t useful. There’s no point in having it around. I suspect this is a good idea, but it raises some red flags for me. People may not want to fix tests if they don't feel like they have time, or fixing tests will help their promotion (i.e. culture). Of course if you have good engineering culture, this is probably a useful sign…

I think maybe the point of that quote is that failing tests are just noise. Either fix them or remove them. If I try to run all tests for some pull request and 20% fail, what does that tell me? In the case where we regularly have some number of continuously failing tests, it doesn't tell me much. Did my PR make it worse? Did my PR make it better? Having continuously failing tests definitely doesn't add value and explicitly makes things harder to reason about when looking at test results.

Re: Improving end-to-end test reliability

#14

Post by Jtsummers »

bluGill wrote: post #12
sidlls wrote: post #6

Earlier quoted context omitted.

That's a rare attitude to have in the Bay Area anyway. Unfortunately. Everyone wants lots of unit tests because they run fast and give (roughly) instant feedback. Unit tests paint an incomplete picture. Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.

The problem I have with unit tests is they inhibit refactoring off the API they test. Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a wh…

That stretches the idea of a refactor, though. If a refactor changes the external behavior it's not really a refactor, which is a structural change. Unit tests when applied at an API level (which may be a very small unit or up to the level of a library, but that also stretches the definition of "unit test" depending on the size of the library) are there to ensure that changes to the internals don't impact the behavior.

As soon as you start changing the behavior, you have to change the unit tests. If you're adding behavior, you have to add tests. If you're removing behavior, you remove tests. If you're changing the way a procedure works, you change the related unit tests.

Really, any behavior change requires changes to the tests (whatever level they may be, if you want a high degree of test coverage).

Re: Improving end-to-end test reliability

#15

Post by avensec »

I appreciate the article for visibility as we could always use more knowledge and partners in the Quality Engineering space. Just recognize that these are relatively low-hanging fruit/early maturity concepts in test engineering.

Are you are discovering these for the first time? Great, happy that you are getting exposed! If you read these and think, "we could utilize these concepts with our engineers(test or not)," I would encourage you to look at it from an organizational perspective. You may want to add someone to your team(s) with these skillsets. Most automation testers understand these concepts well and can help you on the next-level maturity items.

Re: Improving end-to-end test reliability

#16

Post by wereHamster »

vthommeret wrote: post #11

I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it: 1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the…

Have you used Cypress before? If yes I'd be interested in a comparison from your perspective.

Re: Improving end-to-end test reliability

#17

Post by mgkimsal »

Jtsummers wrote: post #14
bluGill wrote: post #12

Earlier quoted context omitted.

The problem I have with unit tests is they inhibit refactoring off the API they test. Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a wh…

That stretches the idea of a refactor, though. If a refactor changes the external behavior it's not really a refactor, which is a structural change. Unit tests when applied at an API level (which may be a very small unit or up to the level of a library, but that also stretches the definition of "unit test" depending on the size of the library) are there to ensure that changes to the internals don't impact the behavio…

And without relatively comprehensive tests, you can't ever tell if a 'refactor' (in the structural sense of the word) actually worked. Did you change internal implementation code without affecting the consumer output? Without tests, you can't reliably tell with a high degree of confidence.

Re: Improving end-to-end test reliability

#18

Post by vthommeret »

wereHamster wrote: post #16
vthommeret wrote: post #11

I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it: 1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the…

Have you used Cypress before? If yes I'd be interested in a comparison from your perspective.

I did, but only very briefly. I originally wasn't looking for an E2E tool but was evaluating another tool for a different problem (Nx) which included Cypress as part of its opinionated defaults.

Cypress was a surprisingly nice experience as well and led me to research other modern e2e tools. Most of the points above can be compared against Cypress — Playwright supports parallel execution of tests within the same file on the same machine, which Cypress doesn't, and so is much faster. Cypress doesn't use modern async / await syntax. Due to its architecture, Playwright can test across tabs, work with iframes easily, which Cypress can't.

The UI for Cypress's developer tools is nice, but... as I said, Playwright's tracing UI is really excellent and the documentation is also really well done. This is also a personal thing, but I trust tools that came out of browser teams (Chrome) to emulate browsers in a more efficient way, e.g. spinning up cheap, isolated browser contexts in Chrome, the details of waiting for an element to be ready, etc...

Another post on this: https://alisterbscott.com/2021/10/27/five-reasons-why-playwr...

Re: Improving end-to-end test reliability

#19

Post by avensec »

8organicbits wrote: post #7

> When an E2E test is failing consistently and nobody cares to fix it, that means the test isn’t useful. There’s no point in having it around. I suspect this is a good idea, but it raises some red flags for me. People may not want to fix tests if they don't feel like they have time, or fixing tests will help their promotion (i.e. culture). Of course if you have good engineering culture, this is probably a useful sign…

One of the reasons we test is for confidence. If we can't trust a test, it isn't providing value. It may give negative value due to the time required to inspect the failure or general erosion of trust in the test suite.

One pattern that we can apply to increase visibility or ownership is stability metrics. If a test must/should be fixed many times can be teased out once you can view these metrics. On failure, display that this test has passed in this configuration for the past x-amount of runs. - Pass the last 100 runs? High likelihood the test is highlighting a bug and must be engaged on. - 95% pass rate in the last 100 runs? It may be time to quarantine this test and add it to the remediation backlog. Your level of acceptable false-positive rates may differ depending on team velocity and suite runtimes.

"How many tests are in quarantine, what is the average time-to-fix, and what direction is this trending" are valuable metrics that we can utilize to find ownership and highlight the technical debt.

As you said, culture around such patterns isn't always there.

Re: Improving end-to-end test reliability

#20

Post by bluGill »

Jtsummers wrote: post #14
bluGill wrote: post #12

Earlier quoted context omitted.

The problem I have with unit tests is they inhibit refactoring off the API they test. Sure if you write your own implementation of "string" or "list" you will probably get the API right the first time - those are commonly used and time tested so everyone knows about what the API should be. However almost nobody is writing them, they come with your language for everyone but a few language implementers, or once in a wh…

That stretches the idea of a refactor, though. If a refactor changes the external behavior it's not really a refactor, which is a structural change. Unit tests when applied at an API level (which may be a very small unit or up to the level of a library, but that also stretches the definition of "unit test" depending on the size of the library) are there to ensure that changes to the internals don't impact the behavio…

Units by definition are not external behavior. sometimes they change external behavior, but there are a lot of changes that make code cleaner without changing external behavior. All too often I've discovered after a few years that I really need to split some unit into two.
Post reply on HN