Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

61–70 of 270 posts

Re: We killed our end-to-end test suite

#61

Post by aidenn0 »

kevindong wrote: post #29

At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a…

Was in a similar situation, and the VP of engineering banned the practice of rerunning failed tests, so flaky tests caused everybody pain. In less than 8 weeks the false positive rate dropped by about 3 orders of magnitude. There's a strong tendency to treat tests as a hurdle to get over rather than to treat them as first-class part of the development process.

Re: We killed our end-to-end test suite

#62

Post by Daishiman »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

It's a fair point, which shows the underlying problem with e2e: if you don't have a BDFL who's willing to fight on this hill, the system will eventually break down. This implies a huge amount of constant friction that I don't believe is sustainable over the long term.

Most engineering organizations don't have "excellent" leadership, and so most orgs are well served by having team dynamics such that they don't depend on that. A bunch of additional integration tests and a bit of formalization of the difficult parts that e2e tests (the sort of async message-passing stuff that has unpredictable bounds) seems like a far better alternative for most orgs.

Re: We killed our end-to-end test suite

#63

Post by fndex »

continuational wrote: post #34

> In our analysis, we figured out that the most frequent category of bugs caught by End-to-End tests was schema violations. Schema violations are pretty much just type errors. Fortunately these can be prevented automatically and with 100% confidence without writing even a single test.

Types are compile checks, they have nothing to do with contracts, having contracts for messages that will transit over a queue are still useful even if you are using a typed language.

Re: We killed our end-to-end test suite

#64

Post by aidenn0 »

Daishiman wrote: post #59
aidenn0 wrote: post #48

Earlier quoted context omitted.

Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production. If an e2e test is not performing reliably not because it can't, but because the test is half-assed, then that needs to be treated as a bug in the test, and the test should not be used to assess t…

> Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production. "Reliably" isn't a binary indicator, but a spectrum of how frequently certain classes of bugs may appear in a system. In the example that you were mentioning, it would appear that the amount of…

> How many man-hours were spent by your manager and staff ignoring the test suite?

I'm not sure what you mean by that. Ignoring the test-suite isn't something that you spend time doing. It was 6ish weeks with a team of 8ish people, so you could say "48 man-weeks" were spent ignoring it, but they were doing other things in that time, not just sitting at their desk proclaiming "I'm ignoring this test."

Once the manager forced someone to fix the test it took less than one man-day to find the bug, and about 5 minutes to fix the bug once it was found.

Re: We killed our end-to-end test suite

#65

Post by QuercusMax »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Yes, this is true in my experience as well.

They are, however, extremely useful when they aren't flaky. A well-built E2E test can be a huge timesaver when debugging interactions between components.

Re: We killed our end-to-end test suite

#66

Post by corpMaverick »

Don't do comprehensive E2E tests. Have a small set of E2E tests to verify the most critical functionality is working. Don't verify every single business rule. There are other techniques that help you reduce risk:

- Good unit and integration tests.

- Effective Monitoring and alerting.

- Canary releases or Blue/Green releases.

- Continuous integration.

- Continuous delivery.

- Ability to safely rollback the more recent release.

Re: We killed our end-to-end test suite

#68

Post by bcrosby95 »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

I have a friend that works on a team whose whole job is writing e2e tests. Before them the tests were slow, buggy, and couldn't be ran in parallel. Now they can be ran in parallel and there are few-to-no false positives.

There's still challenges with this model (such as tracking changes on other teams, helping ensure that UIs are testable), but it seems to have worked out much better for their company than expecting every developer to write and maintain them.

Re: We killed our end-to-end test suite

#69

Post by NewEntryHN »

drewcoo wrote: post #13

E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.

Which can't be proven by E2E either, and which you don't need to prove.

Re: We killed our end-to-end test suite

#70

Post by deckard1 »

msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

> you can't say end-to-end tests are not worth it

You can, actually.

But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more.

They will either slither back into their office, or (most likely) they will insist that developers write E2E in addition to their current workload of writing unit tests, writing the actual code, and all of the other overhead (pull requests, approvals, JIRA ticket maintenance, interviews, etc.)

Developers are expected to pay the costs of E2E with no impact to the business.

What managers do not understand and has been my experience for many years now is that E2E is at least 30% of the cost of development. And that's probably low. I recall certain features where E2E took probably 200% or more time to get working. Because, unlike most unit tests, writing E2E tests is nontrivial. You may have to invent entirely new techniques and apparatus just for a single test.

If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it. Not every bug is critical. In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.

Post reply on HN