Write tests. Not too many. Mostly integration

Nit: liveness is about reaching 'good' states over the lifetime of your computation. If your invariant is violated, even if it's a multistate invariant, it's still a safety error. Liveness would be something like "x is eventually true", or "the program always terminates."

It's not just specifications we need. We also need better tests. "Better" here doesn't mean "integration" or "acceptance", it means things like "fuzzing through contracts" or "comparing snapshots" or "rules-based state machines". Testing is vast and we're not very good at it.

Honest question: Are most of your unit test failures due to bugs or due to refactoring (e.g. changing APIs)?

Most people I know who do unit testing have mostly the latter (easily over 80% of the time). At that point one feels like they are merely babysitting unit tests.

If unit tests have such high false positives, how useful are they?

Note I'm not saying that it's not possible to write unit tests that are relatively immune from refactors. But it is more challenging and, in my experience, quite rare to find a project that writes them this way.

It's a lot faster to catch bugs when the tests run in your local development environment or in the CI pipeline than to wait until Q/A (hopefully) catches them in end-to-end integration testing and sends it back to you for rework.

The other advantage I've noticed is that the tests can serve as an additional form of documentation. New team members can look through the tests to see how code is supposed to work from the examples there. For some web API projects I've even been able to generate the documentation from the test cases.

I've tried various testing strategies over 15~ different companies in all sorts of environments, and unit tests are the only thing that really work (IF you can convince the team to do it...and that's a big IF).

The article starts with a point I agree with: the lower in the pyramid, the cheaper the tests but the lower the confidence level they bring. That's true.

Where I disagree is how much the difference on confidence and cost are.

I can bang out 500 unit tests faster than I can do just a few E2E tests in most large apps. They require almost no trial and error, no real engineering (I feel strongly that abstraction in unit tests is bad), and all around are so easy to write, I don't mind if I have to toss out 150 of time when I make a significant refactor.

E2E tests are amazingly brittle and require a careful understanding of the whole system. They're impossibly expensive to write. They're the only thing that tells you that stuff works though. So you want at least a few of these.

Integration tests are just flat out awkward: you need understanding of a significant portion of code you did not write or touch, they often require complex fixtures (because your test will go through several code paths and might depend on a lot of arguments), they're slower (because a lot of code run), and while you don't throw them away when changing implementation details (unless they involve side effects), you still throw them away when refactoring or changing public interfaces. I've worked with a lot of people who were very vocal about these being so much better, then in the same breath complain that they spent all day writing integration tests.

There's an exception here which is acceptance tests for libraries, especially when doing a full rewrite: the tests that tell you public interfaces used outside of the current context work (as opposed to public interface of objects used in the implementation). Eg: if I was to test lodash or react, that's how I'd do it.

Unit tests to be are about a lot more than "is this change breaking my code". And if that's all you care about, you're missing a big part of the point.

If you have 3 units, A, B and C. A calls B which calls C. If you have a test for A in the context of B, a text for B in the context of C, and a test for C, and they all pass, you know that A + B + C will work. But when writing the tests, you only had to care about itty bitty tiny pieces of code, which made things super cheap.

Then you get other huge benefit: the quality of the entire code base is higher (a side effect of it having testable interfaces all across), the reasoning behind each piece of code is explicit (no one wrote a function that work and they 're not sure why, else the test would be very hard to write), you automatically have a document representing "intentions".

And yes, if you change a module, even if its not expose to your customers, the public interface of that module has tests and the tests will break. But they usually take nothing but a few minutes (often a few seconds) to write. They're cheap enough to be disposable.

And once you have 80%ish unit test coverage, you actually have a very high confidence level. I've gone through major refactoring of multi-million line of code apps with almost no bugs on pure unit tests. You't think the 20% of untested code would be a source of bug, but statistically, that's just not how it happens.

In term of person-hour to ROI, pure unit tests just straight up win out.

The reason software engineers fight back to hard against them is that they're brain dead and repetitive to write, and they can't resist overengineering. "This is such a simple test for such a simple piece of code, why should I test it?!". That's the point. All unit tests should be like this.