Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

21–30 of 270 posts

Re: We killed our end-to-end test suite

#21

Post by asdfasgasdgasdg »

msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

I can hardly think of a situation where I'd want no end to end test.

I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startup time down.

For example, I work on a system that builds text indexes on an underlying database management system. We take an input mutation with logical changes and then use that to determine what additional index updates are required. This all happens automatically when our users write.

There are two ways to test this. The old way was that we instantiated the top level class that did the changes and manually constructed mutations that look like user mutations. Then we examined the mutations produced by our top level class.

I recently converted this test to use the public write and read apis of the database to instead write data to a test instance and then check that the index contents was as expected. The public api is more stable than our private one and is resilient to internal refactorings. It's also more amenable to ad hoc queries of the type you generally do in tests. And it ends up not being much slower, since our test for various sad reasons still had to start the database engine even though it was mostly unused.

All in all, I was able to make the test faster (3 minutes -> 1.5 minutes) and less brittle, while using less code and getting more coverage of what we actually care about. I think wins like this are commonly available when moving from unit to end to end testing, as long as you keep system startup time down.

Re: We killed our end-to-end test suite

#22

Post by coryrc »

commandlinefan wrote: post #9
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

> you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...

Tests are never 100%, customers will always find issues. They made the choice to allow slightly more bugs to production than before, not to go from never having bugs in production to having some.

Re: We killed our end-to-end test suite

#23

Post by exdsq »

Ahhh I've made this mistake before. You can't test every input/output while also losing the ability to accurately depict stateful user flows. Even Fintech projects I've worked on at the >£10Bn daily volume mark combining formal methods, mathematical proofs, property based tests, fuzzing, model based testing, etc.. still caught issues pre-production via good old end-to-end tests.

Good luck nubank :)

Re: We killed our end-to-end test suite

#24

Post by makeitdouble »

commandlinefan wrote: post #4

> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated te…

A complicating factor is that teams usually start with E2E when it’s simple, and only move to other approaches when the systems have become way too complicated.

At that point I don’t know if there is any specific strategy that effectively catches a lot of bugs, short of sending to production and monitor the effect. At a company we just called those tests “sanity checks”, and the goal was just to make sure the most basic use cases would still work, and nothing more.

Re: We killed our end-to-end test suite

#26

Post by Daishiman »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived".

E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Re: We killed our end-to-end test suite

#28

Post by daxfohl »

FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc.

IMO these seem to be more of a CYA thing for managers. When a bug does get to production, you need to have something to point to. (And in my previous company of course they're scrambling now to make a big E2E testing platform). But I'm not convinced they're actually worth the effort.

Edit: actually maybe I think they're worth the effort, if for no other reason than when bugs do go to production, executives tend to start micromanaging if you don't have something to point at. That can be worse than dealing with flaky tests. But what I'm not convinced of is whether they actually reduce the number of bugs that go to production.

Re: We killed our end-to-end test suite

#29

Post by kevindong »

At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a flaky test) up to multiple hours since everyone would rather try to diagnose/hotfix the issue rather than revert their code to unblock the pipeline.

With that being said, a full run of the E2E suite at $PRIOR_JOB took very, very low double digit minutes so it wasn't that expensive. Rerunning a handful of failed tests took single digit minutes so it wasn't too terrible.

Re: We killed our end-to-end test suite

#30

Post by anamexis »

commandlinefan wrote: post #4

> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated te…

> but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal

But, have you seen contract testing fail to catch problems that E2E tests did catch?

I think both of them end up tending to be regression tests a lot of the time.

Post reply on HN