Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

121–130 of 270 posts

Re: We killed our end-to-end test suite

#121

Post by eyelidlessness »

I’m generally not fond of e2e or even integration testing. At least, I prefer to keep them to a minimum, and use other tools to ensure units interact as expected.

That said, where e2e tests may be valuable but costly as described in the article, it occurs to me that narrower integration tests which invert responsibility may be better. Which is to say:

- Given Service A

- Given Service B which depends on Service A

Integration tests of Service A may provide more value if implemented in Service B. It’s SB, after all, which understands the behavior it expects from SA. (If they’re mutual dependencies, of course the inverse applies as well.)

Of course, this highlights (at least for me) why integration tests should be limited in scope. If both services are well tested at the unit level, you will probably end up with a lot of redundancy between their reciprocal test suites. But at least at the idea level, this feels like a better compromise than expecting Team SA to anticipate all of the subtleties Team SB might have in mind.

Re: We killed our end-to-end test suite

#122

Post by x0x0 »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

Our solution was two test suites.

End-to-end (which I will fight for being the highest value test site, and it's not close) had no external dependencies.

And a separate test suite that touched external services, split into two components: one that tested our integrations, typically against a remote testbed (if the 3rd party was competent enough to have such a thing), and a second chunk that attempted to see if remote api behavior had changed. Which it does with annoying regularity.

Re: We killed our end-to-end test suite

#123

Post by CobrastanJorji »

eyelidlessness wrote: post #121

I’m generally not fond of e2e or even integration testing. At least, I prefer to keep them to a minimum, and use other tools to ensure units interact as expected. That said, where e2e tests may be valuable but costly as described in the article, it occurs to me that narrower integration tests which invert responsibility may be better. Which is to say: - Given Service A - Given Service B which depends on Service A Int…

What are the tools you do use to ensure units interact as expected?

Re: We killed our end-to-end test suite

#124

Post by hinkley »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them.

It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth.

Often E2E tests have such sunk costs involved that they materially affect the project roadmap.

Re: We killed our end-to-end test suite

#125

Post by ezekiel68 »

aidenn0 wrote: post #38
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

Flaky tests are an indication of non-determinism either in your test or your system.

Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

Re: We killed our end-to-end test suite

#126

Post by jdsleppy »

evandwight wrote: post #118
kodablah wrote: post #33

Earlier quoted context omitted.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.

As one example, Django handles test parallelism by creating N test databases (on the single test database server) and dividing tests into N runners. https://docs.djangoproject.com/en/3.2/ref/django-admin/#envv...

You could also have multiple Docker containers running DBs.

Re: We killed our end-to-end test suite

#127

Post by shakezula »

hinkley wrote: post #124
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

> Often E2E tests have such sunk costs involved that they materially affect the project roadmap.

I've seen this first hand. When the e2e tests take hours to run, are flaky on a good day, and are only really understood by one or two people on the whole team, they can be a major roadblock to new features or even just moderate refactors.

Re: We killed our end-to-end test suite

#128

Post by ahuth »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> If you maintain speed and strict curation of such a suite

This seems to be the hard part. Any tips for maintaining speed and strict curation, especially at scale (in terms of developers)?

At the very least, it seems that E2E tests are a tool that's easy to misuse. Not sure of the best way to mistake-proof it.

Re: We killed our end-to-end test suite

#129

Post by mirekrusin »

So they ditched e2e in favour of something that average monorepo checks statically ie throught typescript? Then mocked functional tests and called it a day?

The problem probably started when they put themselves in this microservice plague setup where they can't spawn simulated environment in ci anymore. As it turns out running system aka deployment on environment is a monolithic expression of microservice spaghetti.

As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period. Taking aside fact that it inflates test runtime more than decade of mcdonalds diet - if you find yourself in setup with flaky tests, you should ask why they are flaky and amend setup so test is expressed as non-maybe-flaky, normal test. Forbit flakiness, there is no such thing as passed flaky test - those are just shitty tests.

Re: We killed our end-to-end test suite

#130

Post by hinkley »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have a Rule of 8 for the testing pyramid that has been roughly stable across three programming languages.

Each level you crawl up the testing pyramid increases run time for good tests by a factor of 6-10. If your functional tests are taking more than 10 times as long as your unit tests there is something wrong that is worth investigating. Usually I set a default “slow” time equal to multiples of 8 over a good unit test and round off to a whole number to invite fewer questions.

But it also means that if your unit tests are running in 10ms apiece, your integration tests should run in about 640ms and your E2E tests in under 5 seconds. Getting most people to make 3 second end to end tests is at least as hard as getting them to push them down the stack.

You need more tests as you go down, but that generally takes about a 5:1 ratio, meaning you still get a 30% improvement in run time for every test you can push down, and sometimes we are using end to end tests to do unit test work, which is going to be 4 to 500 times faster depending in how many cases were really missing in the unit tests.

Post reply on HN