We killed our end-to-end test suite

In summary they noticed that their e2e suite mostly caught integration errors where clients and servers had incompatible schemas for the data exchange.

The novelty is that they found a much faster way to identify this kind of errors by collecting and comparing the client and server side schemas statically without even running the code.

This is a great optimisation, but it did not remove all defects so they still need to define tests that validate actual application behaviour against the business rules.

> I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

I replied in two other places on this thread before seeing this comment. It's very true. Since tests don't get shipped to customers, tests don't get the same level of effort. But when your tests are known to be of poor quality, people stop trusting them, and when people don't trust the tests, they stop adding any value.

I don't quite agree. A more precise title would have been:

"We Disentangled our E2E Tests"

If I understand correctly - what seemed to have happened is that they separated the lower level data-coupling testing and their higher level testing with contracts and acceptance testing respectively. Each of those layers don't need to know of each other, so this was a separation of concerns AKA simplification.

Why not both?

Beyond unit test, we have docker compose spin up our service(s) and its dependencies. If those dependencies have too big a web, we may point at a staging instance or a fake server, but we routinely will spin up dependencies that will run a local kafka and zookeeper for them to run, are backed by mysql and redis, etc.

We then test our service at its incoming edges (feed its incoming queue or call its endpoints) and verify its output (via logs, metrics, and sinks).

We also have end to end tests that exercise our our services from the customer's point of view, but take place in our staging environment. These do suffer from many of the points the article points out, but we run these tests concurrently, and, when not flaky, can pass in 10 minutes.

We are addressing flaky tests by addressing their root cause: flaky services in staging. We are expecting teams to have mature monitoring of services in staging and tying improvements directly to flaky failed tests. We are also improving traceability so a failed test is easier to debug to understand if it was a failed service request somewhere in the stack.

> Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production.

"Reliably" isn't a binary indicator, but a spectrum of how frequently certain classes of bugs may appear in a system.

In the example that you were mentioning, it would appear that the amount of effort needed to maintain the e2e test suite was simply not worth it. How many man-hours were spent by your manager and staff ignoring the test suite? How critical was the bug (it would appear not much)? How much effort would have to be dedicated to get the e2e suite working will that won't be spent doing other classes of tests or feature development?

I'm not saying a well-maintained e2e suite doesn't work well or help to catch a lot of interesting production bugs. But I am saying that I think that for the vast majority of systems it's just not a good use of your time. Save your efforts and put more thought into the system design to avoid certain theoretical classes of errors and devote the rest of your time to better integration tests and that will likely serve more orgs better.