Buried ten feet deep in the article - they retired E2E tests and introduced "acceptance" tests, which are more efficient E2E tests that they still run on critical code. But I guess "We Renamed Our Eng-to-End Test Suite" isn't a very good blog post.
We killed our end-to-end test suite
11–20 of 270 posts
Re: We killed our end-to-end test suite
#12> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated te…
It says the majority of the problems caught in E2E were due to changed contacts, so it makes sense to have a dedicated test type for those errors. If the remaining caught bugs are few enough and their customers are willing to suffer occasional unavailability, just let them be caught in production and save that expense.
Re: We killed our end-to-end test suite
#13E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs.
https://en.wikipedia.org/wiki/Argument_from_ignorance
Kudos to Nubank for whatever combination of logic and bravery led them to their decision.
Re: We killed our end-to-end test suite
#14I have experienced all the same problems that they outline. E2E tests require a huge number of human-hours to maintain, they're difficult to debug when they fail, almost always false positives, and bugs still get through anyway. But for many situations, there doesn't seem to be a better solution.
For most early stage startups, it seems that time would be better spent optimizing your deployments, rollbacks, and real time metrics so you can maintain a high velocity and roll back quickly when you make a mistake.
For more safety critical systems, the cost of maintaining E2E tests needs to be built into the total engineering cost for the project. It's a hidden cost that is often way bigger than you'd expect.
Re: We killed our end-to-end test suite
#15Contracts are basically unit tests for whatever size of unit you're testing. How do you capture all the dynamic behaviors of a system without some sort of end-to-end test? Delayed timers, queues fill, missed interrupts, locks are held for too long, dead lock, live lock, priority inversion, dropped messages, out of order issues, etc. These things are not captured by contracts and are often exactly why the end-to-end tests were flakey in the first place.
Re: We killed our end-to-end test suite
#16E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.
[deleted]
Re: We killed our end-to-end test suite
#17E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.
Catching all the bugs is generous. You won't be able to prove...basic functionally when all the components are deployed in production
Re: We killed our end-to-end test suite
#18Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue.
Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds trite, but if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end (which also happen to be the ones customers see making the e2e suite a decent place for high level regressions assuming you maintain test performance of course). This is especially true in environment-specific scenarios.
Re: We killed our end-to-end test suite
#19This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…
They didn't say E2E aren't worth it (they actually said it worked for them early on)
Their point is that it's not scalable as they think contract testing is
Re: We killed our end-to-end test suite
#20Almost all the bugs I’ve ever seen have been an integration or configuration issue. Race conditions are a common example of this. Even the small bugs usually involve 2-3 “units.” End to end tests and integration tests, manual or automated, are really the only way to catch these