Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

151–160 of 270 posts

Re: We killed our end-to-end test suite

#151

Post by gravypod »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

One of the things that I love about Bazel is it thinks of a binary that obeys a contract as a test. This means you can have things like `sh_test` which just runs a shell script in a sandbox and gives you all of the benefits Bazel has normally for test execution. You get automated caching, parallelization, and remote execution of tests for free.

A great talk about this: https://www.youtube.com/watch?v=muvU1DYrY0w

You can often get situations where integration tests (that cover large features) take less than 30 seconds, only ever execute your tests when it is possible for the outcome to change (a dep has changed), and you can run your tests on a fleet of machines rather than one laptop or CI runner.

Re: We killed our end-to-end test suite

#152

Post by jseban »

bcrosby95 wrote: post #68
kodablah wrote: post #33

Earlier quoted context omitted.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

I have a friend that works on a team whose whole job is writing e2e tests. Before them the tests were slow, buggy, and couldn't be ran in parallel. Now they can be ran in parallel and there are few-to-no false positives. There's still challenges with this model (such as tracking changes on other teams, helping ensure that UIs are testable), but it seems to have worked out much better for their company than expecting…

Yeah but what's the point then, in that case you can just take back the old QA team and delete the gazillion lines of e2e test code and save yourself the liability of all that complexity. If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

Re: We killed our end-to-end test suite

#153

Post by jrochkind1 »

amw-zero wrote: post #146
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have.

What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Re: We killed our end-to-end test suite

#154

Post by hinkley »

hinkley wrote: post #134
shakezula wrote: post #127

Earlier quoted context omitted.

> Often E2E tests have such sunk costs involved that they materially affect the project roadmap. I've seen this first hand. When the e2e tests take hours to run, are flaky on a good day, and are only really understood by one or two people on the whole team, they can be a major roadblock to new features or even just moderate refactors.

That definitely happens. The E2E tests tend to make assumptions about how the app works (encode not just the requirements but also the architecture) and some features change the design. In order to add this feature we have to fix dozens of other tests. I’ve seen people on multiple projects team up to fix these, take over a day working together, and still not be done. They always try to tweak the tests but the test as…

I should add as well: after that day, day and a half working together on old tests, those engineers look beaten down. They are not having a good time. It’s miserable work.

It must be some sort of Stockholm syndrome that people in this state still defend the tests. Even after they’ve invested more time and energy into fixing them than we ever would just manually testing that part of the code in perpetuity.

Re: We killed our end-to-end test suite

#155

Post by aidenn0 »

Jtsummers wrote: post #93
aidenn0 wrote: post #91

Earlier quoted context omitted.

If people just started throwing random sleeps into tests, I think management would shit a brick. Do people throw random sleeps into production code to fix bugs where you work as well?

Not GP, and fortunately not often, but I have seen that done to overcome race conditions. I pushed for it to be corrected by using a proper design. That was a stupidly hard fight, though.

My pet peeve is people sprinkling C's "volatile" keyword in places. Since doing so inhibits many optimizations, it changes the timing and can make race conditions appear to go away.

Re: We killed our end-to-end test suite

#156

Post by grumple »

mattnewton wrote: post #149
amw-zero wrote: post #144

Earlier quoted context omitted.

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are probably only a handful of completely controlled environments on the planet and even those are suspect.

Test suites are sort of an eventual consistency problem themselves...

Re: We killed our end-to-end test suite

#157

Post by dcow »

amw-zero wrote: post #146
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.

Re: We killed our end-to-end test suite

#158

Post by mabbo »

This is a very bad list of complaints and it actually makes me angry to read it.

> Engineers had to wait more and more to get feedback from this long-running suite

So speed up your tests. Run them in parallel. Find better frameworks for running tests.

> Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative;

Fix your flaky tests! Why anyone just accepts that "Oh, sometimes that test fails and we have to restart everything" is beyond me. Root cause the problem and FIX IT.

> Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge;

Tests should not rely on pre-existing state. Have a setup phase for each test that creates new data in the state you want it to be in. As it makes this data, also note down a reference to it with a Time To Live so that a follow-up process can clean up this unneeded data.

> Test failures were very hard to debug

That's not the fault of the tests, that's the fault of a complex system that is hard to debug. Improve your tracing between services.

> Queueing of commits in the End-to-End suite resulted in less frequent deployments

There are well known solutions to this problem. Lots of companies have overcome this already.

> Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug

If your tests have false-positives, you need to adjust your tests. Accepting that the test failed but there isn't a problem, and then not fixing the reason the test failed means that you don't have reliable tests.

> Bugs were still being found in production

Bugs will always make it to production. But after you fix a bug, you write a test so that this bug cannot happen again. Over time, the number of possible bugs that can make it to production shrinks.

And lastly:

> The main difference to the old E2E is that they encompass only a subset of services and don’t require spinning a production-like environment (the services run in memory on a single JVM and HTTP/Kafka communication is replaced by in-process communication). They are used in specific flows that we find too critical to only rely on Contract Tests.

Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra?

Re: We killed our end-to-end test suite

#160

Post by aidenn0 »

amw-zero wrote: post #144
ezekiel68 wrote: post #125

Earlier quoted context omitted.

Flaky tests are an indication of non-determinism either in your test or your system. Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I mean the exact output given a certain set of inputs may be slightly different due to asynchrony, but given a set of inputs, there should be a finite set of correct outputs and check for those.

To use a stupid example: if listAnimals returns [cat, dog, mouse] some of the time and [cat, mouse, dog] other times, if your passes on the former and not the latter, then your test is broken and you should fix it. If it sometimes returns [cat, dog, mouse, tree] then your system is broken and you should fix it.

Post reply on HN