Improving end-to-end test reliability
frantic.im
Improving end-to-end test reliability
1–10 of 24 posts
Re: Improving end-to-end test reliability
#2No post body was provided.
Re: Improving end-to-end test reliability
#3I love learnings on automated testing. From the perspective of someone who isn't used to TDD or even just building many tests, maintaining E2E tests often seems extremely cumbersome. I wonder if I'm just missing out on the best practices, or if the tooling simply hasn't evolved enough yet.
Re: Improving end-to-end test reliability
#4I love learnings on automated testing. From the perspective of someone who isn't used to TDD or even just building many tests, maintaining E2E tests often seems extremely cumbersome. I wonder if I'm just missing out on the best practices, or if the tooling simply hasn't evolved enough yet.
The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem.
Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your own code. Adding tonnes of unit tests to that is sort of missing the big picture - you need to test it all works together as a user would.
Re: Improving end-to-end test reliability
#5The title should mention this article is from 2019. I wonder if Facebook testing practices have changed since then.
Re: Improving end-to-end test reliability
#6I love learnings on automated testing. From the perspective of someone who isn't used to TDD or even just building many tests, maintaining E2E tests often seems extremely cumbersome. I wonder if I'm just missing out on the best practices, or if the tooling simply hasn't evolved enough yet.
The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem. Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your o…
That's a rare attitude to have in the Bay Area anyway. Unfortunately. Everyone wants lots of unit tests because they run fast and give (roughly) instant feedback. Unit tests paint an incomplete picture. Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.
Re: Improving end-to-end test reliability
#7> When an E2E test is failing consistently and nobody cares to fix it, that means the test isn’t useful. There’s no point in having it around.
I suspect this is a good idea, but it raises some red flags for me. People may not want to fix tests if they don't feel like they have time, or fixing tests will help their promotion (i.e. culture). Of course if you have good engineering culture, this is probably a useful signal for tests to remove.
Re: Improving end-to-end test reliability
#8I love learnings on automated testing. From the perspective of someone who isn't used to TDD or even just building many tests, maintaining E2E tests often seems extremely cumbersome. I wonder if I'm just missing out on the best practices, or if the tooling simply hasn't evolved enough yet.
The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem. Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your o…
Yep same. For libraries unit tests are great. For applications though I feel the most value writing integration tests and e2e tests. That's what helps capture the biggest user-facing bugs.
Re: Improving end-to-end test reliability
#9Earlier quoted context omitted.
The payoff is much higher imo though. Of all the tests we do, e2e catches by far the most problems. Indeed the biggest mistakes I've made often are me thinking tests are flakey 'because e2e' when in reality they are showing a glaring problem. Especially in mobile/web applications where you are often consuming loads of services/libraries/sdks, some in house, some external, you are often running a tiny amount of your o…
That's a rare attitude to have in the Bay Area anyway. Unfortunately. Everyone wants lots of unit tests because they run fast and give (roughly) instant feedback. Unit tests paint an incomplete picture. Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.
> Too little attention given to integration tests and end-to-end tests leaves systems exposed to critically bad edge case bugs.
From my POV and experience, the middle ground is often what people refer to as 'integration' tests. Testing (without a browser), hitting endpoints/urls with known payloads and getting expected results catches errors with assumptions made about the interaction between various individual libraries.
At least in the web app world, my views are:
1. Testing the individual libraries gets you one layer of confidence. 2. Testing the interaction of those, usually via URL endpoints as various identities, gets you another layer of confidence. 3. Testing with E2E exposes primarily UI/JS problems.
When the first 2 are strong/solid, you can focus troubleshooting problems in #3 at the client/JS level first. It's not always the case, but it can help reduce concerns about "is this a back-end issue?".
I've been (slowly) trying to write more js component tests (in one case, with jest and vue), as it makes it easier/faster to test many permutations of input/validation/etc all at once. It's yet another 'confidence' area such that, when there are E2E tests, I can narrow down focus even more.
On a couple projects I've been on the past few years, we've found very few problems via E2E tests alone, mostly because there are so many back-end unit and integration tests. The E2E issues that are found are often UI-only (error state changes not rendering, sometimes perf issues, etc).
Re: Improving end-to-end test reliability
#10This article is spot on. For $LargeNetworkHardwareVendor we maintained three different automation test frameworks for end-to-end testing. Our tests were more abstract functions that were given arguments for a particular test case. Those were then made into collections of tests that could be re-used. A configuration file allowed QE to build new test cases without programming knowledge. QE would write configs and occasionally one or two of them that could code would modify the test framework. All the tests ran in a scheduler from clusters of test-running manager-servers against globally distributed labs of hardware. While teams did have unit tests and functional tests, the end-to-end test was king (and necessary given the multiple levels of interface for that gear)
A lot of reliability in that system came from being able to quickly iterate on different levels of the system. The easier it was to solve a failure where it's happening, the more likely your bugs can be fixed quickly, so you have a healthy system (as opposed to suffering from entropy and tech debt)