Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

141–150 of 270 posts

Re: We killed our end-to-end test suite

#141

Post by cratermoon »

aidenn0 wrote: post #61
kevindong wrote: post #29

At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a…

Was in a similar situation, and the VP of engineering banned the practice of rerunning failed tests, so flaky tests caused everybody pain. In less than 8 weeks the false positive rate dropped by about 3 orders of magnitude. There's a strong tendency to treat tests as a hurdle to get over rather than to treat them as first-class part of the development process.

At one place I consulted, the fte lead ignored flaky tests and attributed failures to the tests being wrong.

A few months later...

The code that was failing intermittently was found to be using floating point types for money. Yeah, I'm gonna wanna fix that.

Re: We killed our end-to-end test suite

#142

Post by catern »

eyelidlessness wrote: post #121

I’m generally not fond of e2e or even integration testing. At least, I prefer to keep them to a minimum, and use other tools to ensure units interact as expected. That said, where e2e tests may be valuable but costly as described in the article, it occurs to me that narrower integration tests which invert responsibility may be better. Which is to say: - Given Service A - Given Service B which depends on Service A Int…

Yeah, I think the easiest way to test that Service A is meeting its API obligations is to send some requests to Service B. Hyrum's Law means that that's the only way to really test the important aspects of Service A's API.

And I think this can be generalized into a general philosophy of using your users as a test suite for your API: http://catern.com/usertests.html

Re: We killed our end-to-end test suite

#143

Post by mst »

gls2ro wrote: post #113
mst wrote: post #109

Earlier quoted context omitted.

I think what they mean, basically, is "e2e testing absolutely everything was becoming a nightmare, so we've now switched to 'contract based testing' - effectively 'unit testing where the microservice is the unit granularity' - plus some e2e style testing for critical paths where it's still valuable enough to justify all the extra effort".

Ok. In this case it is still e2e. In my head e2e is never a strategy to test everything. I also thing they did a good optimizations to offload some cases from e2e go some other form of testing which for should be integration testing. If they dont do integration testing then a lot of possible bug cannot be found. Just testing the input and output of each microservice is not enough. But I will watch the video as maybe…

Yes, I said it was still e2e.

What they meant - I think - was they moved away from "a primarily end-to-end integration test suite" to "service-as-black-box unit-ish testing" plus "focused end-to-end integration testing for critical paths".

Re: We killed our end-to-end test suite

#144

Post by amw-zero »

ezekiel68 wrote: post #125
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

Flaky tests are an indication of non-determinism either in your test or your system. Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

Re: We killed our end-to-end test suite

#145

Post by amw-zero »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

There's basically no value in having tests against third-party code anyway, because all the test is going to do is tell you that they broke their interface. And by then, production is already broken.

Re: We killed our end-to-end test suite

#146

Post by amw-zero »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue.

At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There are no magical optimizations. This is something that every project runs into, over and over again.

I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have.

Re: We killed our end-to-end test suite

#147

Post by cratermoon »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The entire article is about the reasons they ditched the test suite and replaced it with a different practice. Does it need to be more specific about the tradeoffs between fixing/rewriting the e2e suite vs. doing something different?

Re: We killed our end-to-end test suite

#148

Post by jseban »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Yeah I agree with you, and it's because writing a test suite to simulate the users and verify (all) the use cases of your system, is really complex. I think the test advocates make it way too easy for themselves when they always just say "the first rule of testing is that your tests should always be fast" See, you broke the first rule, that's your problem! Well.. how do you execute a large number of complex operations and verifications, quickly? There's a how lot of actual practical solutions missing here, and just a lot of obstinate principle belittling "rules" and deflecting from providing an actual solution, which in practice, is hard.

Re: We killed our end-to-end test suite

#149

Post by mattnewton »

amw-zero wrote: post #144
ezekiel68 wrote: post #125

Earlier quoted context omitted.

Flaky tests are an indication of non-determinism either in your test or your system. Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I should be able to test that this usually works though, right?

Re: We killed our end-to-end test suite

#150

Post by mattnewton »

amw-zero wrote: post #145
derefr wrote: post #82

Earlier quoted context omitted.

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

There's basically no value in having tests against third-party code anyway, because all the test is going to do is tell you that they broke their interface. And by then, production is already broken.

I agree this is often the case but disagree it is always; “testing” against the api can be a canary for your new usage of their third party api not working the way you think it does.
Post reply on HN