Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

221–230 of 270 posts

Re: We killed our end-to-end test suite

#221
post #13

E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.

I don’t think it was bravery. I think it was lack of end-to-end test engineering skills. If your tests are flaky then do what you would do with any other software: fix it! If your end-to-end tests are slow then do what you would do with any other software: optimise it! Not having proper end-to-end tests is a massive red flag.

Re: We killed our end-to-end test suite

#222

I have experienced all the same problems that they outline. E2E tests require a huge number of human-hours to maintain, they're difficult to debug when they fail, almost always false positives, and bugs still get through anyway. But for many situations, there doesn't seem to be a better solution. For most early stage startups, it seems that time would be better spent optimizing your deployments, rollbacks, and real t…

Not my experience at all. Having solid end-to-end tests means that I spend 99% of my time adding new features and 1% of my time fixing bugs before deploying. I haven’t had a bug in production for years because of solid end-to-end tests.

Re: We killed our end-to-end test suite

#223

> One of our Sr Staff Engineers, Rafael Ferreira, ran some numbers and applied queueing theory. Is it just me or is queueing theory seemingly misused all over the place in software engineering. I don't know what was applied here, as it's omitted, but I've certainly seen people suggest that doing things like mandating that cycle time is decreased will increase throughput and justifying what they are saying with "queue…

All models are wrong. Some are useful. The problem is knowing if a specific model is useful.

Re: We killed our end-to-end test suite

#224
post #158

This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running suite So speed up your tests. Run them in parallel. Find better frameworks for running tests. > Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; Fix your flaky tests! Why anyone just acce…

Agree. They clearly are lacking in test engineering skills. So the right solution would have been to hire somebody who knows how to do this well. Bad tests are simply bad code. So fix it!

Re: We killed our end-to-end test suite

#225

'E2E' is often impossible as in most ecosystems as things are constantly changing and you will dependencies out of your control you cannot simulate. The key is faking the right dependencies with accurate-enough versions to keep test fidelity and speed, keeping the test svelte and fast enough so it can run before you merge code to eliminate the size of change being tested and thus more easily understand the outputs to…

Test against API’s. Not implementation! Your API’s are supposed to be stable. If they are not then you are doing it wrong.

Re: We killed our end-to-end test suite

#226

At $PRIOR_JOB, it always felt like the full E2E tests approached useless since for every bug successfully caught, it felt like there were ~20 false positives. At which point, everyone (myself included) blamed the tests and just repeatedly reran the tests until they usually passed. Every single failure would halt the pipeline anywhere from 5 minutes (in the case that rerunning the failed test shows that it was just a…

Obviously the E2E tests were really badly implemented. Implementing solid E2E tests is a skill that needs to be learned like any other software development skill. Most developers don’t know how to do it well.

Re: We killed our end-to-end test suite

#227
post #39

I work a lot on compilers and VMs and have written tens of thousands of tests at different scales over the years. Different kinds of tests serve different purposes. Unit tests help you pinpoint errors in the code. They can exhaustively test (only) small components to make sure they are fully compliant. They are a refactoring and development aid to the extent that they are focused (don't involve too many components),…

Agree. I wouldn’t be able to confidently deploy my code into production without solid tests. I haven’t had a production bug for years because of those solid tests catching it early.

Re: We killed our end-to-end test suite

#228
post #135

Earlier quoted context omitted.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

Your comment matches my experience very well. I had the same experience as GP and OP with low-quality e2e tests at my job. I got fed up four years ago, started something new from scratch, and now I'm the BDFL you mentioned, for a bunch of teams working in a common testing framework. The main thing is indeed enforcing high quality standards even when individual engineers aren't very invested. You've identified some go…

I'm someone who had to build some mocked services to do end-to-end testing (well, as much as we can). The stuff I work on involves making two DNS requests (to different providers) and a possible HTTP request (for notification) and these three end-points are not under our control (as far as the department I work in are concerned). The two DNS requests are made concurrently [1] and management wanted to test the following scenarios:

* A returns, then B;

* B returns, then A;

* A returns, B returns late [2];

* B returns, A returns late;

* A returns, B never returns;

* B returns, A never returns.

I had to implement a side channel from the testing program to the mocked DNS servers (because a program like bind is just overkill for this---seriously) to implement artificial delays in the responses. Kind of hard to justify that for a production server (and yes, there is an active bug where B returns but A doesn't and the wrong information is returned, but it happens so rarely in production [3] that it was deemed acceptable for now).

The other component, the notification via HTTP, required ensuring that a notification that wasn't supposed to happen, didn't happen. [4] Again, I had to implement a mock with a side channel to the testing program to inform if it was to expect a request or not, and then report after all the tests were run how many requests were actually made. If the value between the testing program and the mock didn't match, it's an error. Oh, it's also useful to inform the mock what HTTP status code to return for the test. Such fun.

Management doesn't seem to think these mocks are a waste of time, but it seems like you might.

[1] At least for now. In the past, there were cases were we were to only contact A; some cases where we contact B, then maybe A; and some cases where we contacted both. This was done to save money at the time because all queries cost us money.

[2] we have some real time constraints on handling queries from our customers, the Oligarchic Cell Phone Companies.

[3] Excessive KPI logging for the win here.

[4] Proving a negative---lovely. Thanks, management!

Re: We killed our end-to-end test suite

#229
post #28

FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc. IMO these seem to be…

If you have an E2E testing platform, and a bug is found in production, can you reproduce the issue with the E2E testing platform? How easy is it?

Re: We killed our end-to-end test suite

#230

Earlier quoted context omitted.

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…

“In fact, go back 15 years and no one had any tests whatsoever. The world didn't end.” That’s just silly. Of course there were tests 15 years ago. Unit testing has been around since the 1950s Very little would cause the world to end. But lives have been lost and billions of dollars along the way wasted due to improper testing

Not in the same way as a modern CI/CD-pipeline. 15 years ago you had release-approval meetings where a dedicated QA team would present all their manually observed findings in a excel-sheet to management.

E2E is a replacement for, or evolution of, the QA department, not a replacement for unit tests.

Post reply on HN