We killed our end-to-end test suite

> Waiting. Engineers had to wait more and more to get feedback from this long-running suite;

"Our tests are inefficient, not sufficiently parallelized, the setup / tear down of the test environment isn't optimized, and it isn't possible to run only the relevant subset of tests during feature development or bug triage for short feedback loops"

> Lack of confidence. Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative;

"Our tests aren't well written (we have sleep-polling)", "we don't build-in testability into our system (we can't introspect or wait on the thing we care about in the test, so we have massive work arounds)", or possibly worst "our system is flaky and our tests reflect that".

> Expensive to maintain. Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge;

"We haven't spent enough time developing our own tools for testing, so we have tests that are extremely fragile (think copy and paste of massive JSON blobs with comparisons just to check a handful of values)"

> Failures don’t point to obvious issues. Test failures were very hard to debug, specially due to our reliance on asynchronous communication that make it hard to connect the cause of failure (a message not published to a queue) with its effect (changes not made in another system);

"Our system is over-engineered and our service boundaries match our internal structure rather than clean separation in the functions of our APIs. We don't have good visibility because doing any one thing involves massive levels of coordination. We lack proper tracing and aggregation."

> Slower value delivery. Queueing of commits in the End-to-End suite resulted in less frequent deployments;

"Quality is hard and takes time. Let's not do it so we can move fast and break things."

> Not efficient. Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug;

See above about flakiness and fragility. Also, integration tests catching bugs tend to point to really bad obvious bugs. I'd be happy about the one that was caught.

> Not effective. Bugs were still being found in production.

"We still found bugs. This means testing must be ineffective altogether?"

I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

Trying to wrap my head around what is meant by that. I mean I get the second half, but the first half not so much.

Unit tests help you pinpoint errors in the code. They can exhaustively test (only) small components to make sure they are fully compliant. They are a refactoring and development aid to the extent that they are focused (don't involve too many components), quick (run in seconds or less), not too tightly coupled to the code under test (i.e. can change code under test without changing tests), and explanatory (failure output is easy to understand and points exactly at the faulting component). Making good unit tests is an art form. Some people love their mocking frameworks. Personally, I hate them. Mocks are confusing and they try to check behavior rather than input/output. They make refactoring hard because they test behavior rather than results, and they are usually confusing.

Integration tests are about putting one or more systems together to test their interactions. More than just a single unit, we can put services together and test their interface. They can be more exhaustive about testing a component's interface because the combinatorics haven't exploded yet. Because there is a lot more code under test, failures are less explanatory and thus there is more work to investigate these failures. Investments here that help are to make failure modes as helpful as possible. That, too, is an art form.

End-to-end tests are inherently going to be slow. We put the whole system together and run some canned interactions on it. It might be flaky (because large scale, because networks, because OOM, timeouts, etc). End to end tests are generally a bitch to debug, because essentially anything could be at fault...well, anything except the things that are clearly passing their unit tests and integration tests. Which is why you need to have good unit and integration tests, so that you don't need many end-to-end tests.

It sounds from the article like they reduced or eliminated their end-to-end tests and went for more integration tests. That does seem to have paid off. Sometimes tests are slow and bad, and other kinds of tests are better.

I would say though, working now on a system with many, many, distributed moving parts, you do want to at least have some end-to-end tests that make sure everything comes up properly. Nothing like committing a change that passes all the small scale tests and then a component fails to come up because some stupid command-line flag is set wrong. You gotta have tests for anything you could absent-mindedly break.

And all of that testing needs to be a one button push away. You can't have tests that developers don't run, or don't know exist. Personally I like having shell scripts that are checked in and all at least one that does the whole enchilada, even if it is just a wrapper around the build system's or CI's test targets.