Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

81–90 of 270 posts

Re: We killed our end-to-end test suite

#81

Post by gchamonlive »

kodablah wrote: post #45
gchamonlive wrote: post #40

Earlier quoted context omitted.

> if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end Points 2, 4 and 7 from the assessment expose why sometimes this is not achieved even in E2E tests.

Well of course e2e suites are not a panacea. That doesn't support killing them. With regards to flaky or hard to debug tests, those are implementation-specific issues that should not be used dispel the entire concept of e2e testing (and can usually be solved by high code-quality tolerances and tracing respectively).

> That doesn't support killing them.

If you have to wait hours, sometimes days for a queue to run tests that catch 1 bug in 1000 runs and you still end up with bugs in production, I believe this supports killing the current e2e process, if you find ways to guarantee system integrity between services.

I think you are assuming accidental complexity, but hard to debug tests could also be a symptom of inherent system complexity.

Nubank is a gigantic proponent of clojure and is regarded to have high standards of code quality, so I think that we can give them the benefit of the doubt in this aspect.

Re: We killed our end-to-end test suite

#82

Post by derefr »

aidenn0 wrote: post #38
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments.

...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties?

(I know my answer, but I'm curious to hear yours.)

Re: We killed our end-to-end test suite

#83

Post by deckard1 »

aidenn0 wrote: post #61
kevindong wrote: post #29

At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a…

Was in a similar situation, and the VP of engineering banned the practice of rerunning failed tests, so flaky tests caused everybody pain. In less than 8 weeks the false positive rate dropped by about 3 orders of magnitude. There's a strong tendency to treat tests as a hurdle to get over rather than to treat them as first-class part of the development process.

I imagine this would just turn into everyone inserting 10 second pauses on the tests that fail. Which works, but now your suite doubles the run time. Actually turning nondeterministic tests into deterministic ones is... hard. Really hard in some cases. Many devs don't even understand how to get there, even after years of E2E experience.

One place I worked, the E2E suite took a full hour to run. Everyone reran the tests. Merges took a full day in many cases. Management tried to force people to fix broken tests. But they also required new tests on new features. So it was a constant treadmill. There was basically a full mutiny by the end and the company killed off their entire E2E suite.

Re: We killed our end-to-end test suite

#84

Post by gls2ro »

alex-freire wrote: post #76
alex-freire wrote: post #74

Earlier quoted context omitted.

Thanks for the feedback, we didn't want to burry the "acceptance test" complement to our testing strategy in the article. If you're curious how it works we've recorded a webinar about it: https://www.youtube.com/watch?v=wKgDaD5Nie4&list=PLfqo9_UMdH...

And in part 2 about 12 minutes in we show the code to exemplify how our acceptance test suite works (and is very different from traditional E2E): https://www.youtube.com/watch?v=caxpxszueI0&list=PLfqo9_UMdH...

I cannot watch this entire video, so it is possible that you explain conceptually how your acceptance tests are different from end to end test?

I would like to undestand what were the goals of your initial end to end and what are the goals of acceptance testing, and how you define these acceptance testing from test objective perspective.

I assume end to end tests could be described by this definition: "test the functionality and performance of an application under product-like circumstances and data to replicate live settings. The goal is to simulate what a real user scenario looks like from start to finish" [0]

[0] https://smartbear.com/solutions/end-to-end-testing/

Re: We killed our end-to-end test suite

#85

Post by mattwad »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

I always mocked out 3rd party tests in my tests. I've never actually had a problem with some third party changing their API. That's the whole point of a versioned API anyway. I think when people talk about e2e tests, it's more about testing only integration between contracts that you own.

Re: We killed our end-to-end test suite

#86

Post by vemv »

The middle ground that not enough teams are exploring is following the so-called Functional Architecture. If all side-effects are effectively segregated and reified, then one should be able to swap them out for determistic mocks that run instantly.

So you could E2E a distributed system realistically and instantly.

You can still simulate things services being slow, unavailable, etc if the code handling those is expressed as pure logic instead of coupling itself to IO.

Interestingly, in addition to fixed examples, you can perform generative testing over this setup e.g. what happens for various combinations of services being slow/down.

Re: We killed our end-to-end test suite

#88

Post by aidenn0 »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

This system didn't rely on third party services, so "not applicable" I guess?

Re: We killed our end-to-end test suite

#89

Post by [deleted] »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

[deleted]
Post reply on HN