Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

41–50 of 270 posts

Re: We killed our end-to-end test suite

#41

Post by lkrubner »

Test suites will tend to fail the more your system has to work with "outside" data. I recently had a client where their own data was entirely dependent on data drawn from 23 different 3rd party APIs, which meant the bulk of their code was devoted to parsing APIs they had no control over. Those external APIs sometimes changed, and sometimes contained bugs (that is, violations of published contracts).

To talk about this, I use a broad definition of "outside data". If you're a small startup, "outside data" typically refers to data that belongs to another company. But if you're working in a Fortune 500 company, "outside data" can also refer to data coming from an API run by some other division, which is nominally part of "your" company but is affectively independent.

One rule I now offer to my clients: the more your system relies on outside data, the more it is helpful to have run time checks, rather than a test suite. Assuming you run your code on multiple machines or nodes or dynos or instances, you can chose to run the tests on just a percentage of your system, enough to detect problems, but without paying the performance price on 100% of your system.

When a problem in your system is because of a change in an external API, your test suite won't catch it, since your test suite works with dummy data. But run time checks will catch the problem and make debugging easy -- you'll see almost instantly which API call created the problem.

Code written on the JVM has the beautiful property that you can add pre and post assertions on every function, and you can pass a flag to the compiler asking that the assertions either be left in the code or stripped out. This makes it easier to build 2 copies of the code, one with the asserts and one without, and that makes it easier to, again, deploy the code in such a way that only a limited percentage of your code needs to run those run time checks.

Re: We killed our end-to-end test suite

#42

Post by jniedrauer »

drewcoo wrote: post #13

E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.

E2E tests are still extremely limited and let bugs through, unless they also fuzz somehow. But that will make them even more flaky and difficult to debug, costing more time. It's a tradeoff.

Example: You run some number of operations and then batch them. If you run the exact same operations each time you test, you may not catch conflicts between them. Instead, you'd have to run a random number of, and type of, operations. But then test failures would become extraordinarily difficult to reproduce. You'd have to hope that your tests log exactly what the inputs were, and have a semi-efficient way to recreate those inputs locally.

Re: We killed our end-to-end test suite

#43

Post by siliconc0w »

'E2E' is often impossible as in most ecosystems as things are constantly changing and you will dependencies out of your control you cannot simulate. The key is faking the right dependencies with accurate-enough versions to keep test fidelity and speed, keeping the test svelte and fast enough so it can run before you merge code to eliminate the size of change being tested and thus more easily understand the outputs to determine if they represent a false positive or negative and where any problems may be. This also all requires building the infrastructure to spin up a simulated world quickly enough to simulate how the proposed change effects the simulation and then analyze the results which is also pretty hard and can get expensive.

Luckily spending time optimizing often helps test speed and can control cost so there can be a good case to make for it but orgs have to be willing to pour engineering hours into that and engineers need to want to do it vs building new things which is typically more enticing.

Re: We killed our end-to-end test suite

#44

Post by realusername »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Same experience, I'm sure you can design a large e2e test which isn't slow and flaky but for that you need some very very strict set of rules & care. I've personally never experienced one like this though.

Re: We killed our end-to-end test suite

#45

Post by kodablah »

gchamonlive wrote: post #40
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end Points 2, 4 and 7 from the assessment expose why sometimes this is not achieved even in E2E tests.

Well of course e2e suites are not a panacea. That doesn't support killing them. With regards to flaky or hard to debug tests, those are implementation-specific issues that should not be used dispel the entire concept of e2e testing (and can usually be solved by high code-quality tolerances and tracing respectively).

Re: We killed our end-to-end test suite

#46

Post by [deleted] »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

[deleted]

Re: We killed our end-to-end test suite

#47

Post by JackFr »

toddh wrote: post #15

Contracts are basically unit tests for whatever size of unit you're testing. How do you capture all the dynamic behaviors of a system without some sort of end-to-end test? Delayed timers, queues fill, missed interrupts, locks are held for too long, dead lock, live lock, priority inversion, dropped messages, out of order issues, etc. These things are not captured by contracts and are often exactly why the end-to-end t…

System A listens to queue B and handles every kind of message b throws at it. But somewhere, at some point in time, some coder has made the innocuous assumption that B_id's are unique.....

Re: We killed our end-to-end test suite

#48

Post by aidenn0 »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production.

If an e2e test is not performing reliably not because it can't, but because the test is half-assed, then that needs to be treated as a bug in the test, and the test should not be used to assess the quality of your software. Developers (including me!) have a natural tendency to treat bugs in tests as lesser than bugs in the product, but given that bugs in tests will mask bugs in the product, this is a problem.

True story: a new e2e test was failing randomly. For 6 months nobody looked at it because "it was just a flaky test." A manager found out and insisted that someone fix the test, and it turned up the test was fine, it just found a (non-deterministic) bug that had been in the product for over a decade.

Re: We killed our end-to-end test suite

#49

Post by it »

If your system can't be tested end-to-end because of how slow and flaky it is to test that way, doesn't that say something about the quality of your system rather than the tests?

Of course once you have such a system, it's probably the result of years of work by many people and most likely it would be hard to make it faster and more reliable. That is probably why people shy away from doing that, and choose to blame the tests instead.

Re: We killed our end-to-end test suite

#50

Post by zemo »

> Manual changes in our staging environment corrupted test data fixtures

there's a lot here.

Manual changes in your staging environment shouldn't affect your tests, because your tests should be isolated from other environments.

Also fixtures are generally bad. Given some fixture representing an initial state S, a test utilizing this fixture along with some acceptance criteria is essentially testing that given the state S, running the tests executes some transformation T such that the state of the system is now S2; acceptance criteria evaluate S2 against some known-good value to confirm that T is the desired transformation. This is meaningless if the initial state S is not actually reachable by the system. The fixture itself does not prove that S is reachable: that S is reachable is taken as an act of faith.

So how do you determine that the initial state S is reachable by the system? Well, you have some other test, that starts with an initial state of nothing, that performs some transition (generating and inserting random data instead of using a fixture, for example) and gets nothing into the state S. By doing this, you've both created the state S _and_ verified that S is a valid, reachable state. Now you run your second test after the first test in sequence. To run N tests off of initial state S, you replay the initial test that produced state S N times, once for each dependent test. Sure that's a lot of work, but each sequence of testing events can be run in isolation from the others, so they can be run in parallel.

Post reply on HN