Improving end-to-end test reliability

Unit tests fall on the far left, workload tests/E2E tests/testing-in-production fall on the far right.

It turns out that there's no 'wrong' level, there's just different tradeoffs. I've worked at a lot of companies that embraced the realness of E2E tests, but then suffered from the maintenance/performance/diagnosability/instability of those tests. I have colleagues who worked at places that avoided E2E at all costs, and suffered because they would have a green test run, but user scenarios that a simple E2E test would have caught, were completely broken.

IMO there is a lot that can be done to improve E2E testing at most companies, but they definitely have the capacity to add value to your release/testing pipeline.

If you have one or more integrations to external systems, where you cannot control your test data, it becomes much harder to write stable E2E tests.

Some don't have test environments, some have too few. Most don't allow you to setup data easily either way.

You can, of course, mock the external systems, but if they play a large enough part, your tests start looking more like integration tests again, but with the added overhead of something like browser automation.

It's a hard balance to strike.