The Myth of Code Coverage

100% is the way to be. Once you get there, then you can start having useful conversations about how to make the coverage more meaningful. But first, you have to make sure that every line is run at least once without crashing (except for the lines that are supposed to crash things - those you need to verify DO crash things).

Interestingly, once something is designed for testability it is more likely to not have bugs. But saying that therefore you don't need the tests is silly, because without the tests you wouldn't have designed for testability.

> You can always cherry-pick it back later and know that it works.

Exactly, "this is what git is for". I think being uncomfortable tossing tested code is more of a reflection of the temperament of the person saying it than any truism about coders in general. If anything a little bit of a roadblock with test cases can be a good thing. "By deleting this test I am really certain I want to remove this feature".

There's one place in my code base where I have extra tests in there, that I have to replace every time (it's snapshot-testing code-generating code), but that one little roadblock make me feel good because I know that the code that is being emitted is human-readable and easy for an end-user to debug.

> I have to replace every time (it's snapshot-testing code-generating code),

Do you store your snapshots separately?

One note for devs reading this: I think snapshot tests are very cool, but I recommend trying them as a very separate thing than unit tests (if you are not already doing that). I removed all snapshot tests from our unit testing pipeline and it's much better for it. Was very surprised that Jest doesn't have warnings that snapshot testing is a very different thing, and should not be intermixed with unit tests.

Snapshot testing is sort of like code coverage, a great tool for telling you what to look for, and you should make it so that doing snapshot testing is easy, but the snapshots themselves should be stored outside of the main repo/main testing repo, and in their own thing. Not sure if you are doing that or not, but for anyone else looking at snapshot testing, this is a common mistake I see.

> You want to strive to test that logging logs and observability observes?

You're asking this rhetorically, but I often find interesting problems once I start really exercising the latter, like:

- Non-global collectors we forgot to register, or fail to get registered on some code paths.

- Collectors that get registered twice or overlap with other collectors only in some conditions (e.g. connection addresses that are sometimes the same and sometimes not depending on how your service gets distributed).

- Collectors that are not really safe scraping concurrently with the rest of the program; our test suites include race detectors.

- Metrics that don't follow idiomatic naming conventions (can be found via e.g. `promtool check metrics`).

- Metrics with legitimate difficult-to-notice bugs. We had an underflow counting the size of a connection pool because `Close()`ing one twice was explicitly allowed.

> Testing trivial code brings negative value

Agreed in the abstract, but if we were good judges of what code was trivial or not, we'd write a lot less bugs in the first place!

That being said, this is also a discussion about coverage, not assertions. Even if I'm not actually checking any of those things, those code paths (and logging) are still getting exercised via any decent E2E/functional/behavioral test suite.

But can't you identify the signal of the input and output and write small test cases to test just the signal?

Snaps test the signal + noise. They help alert you when either changed, which is helpful if you think you may be missing tests for the signal, but ideally you would just make sure all the signal is tested, and the snaps would be just a later, independent sanity check tool.

Obviously you know the details and often once I learn the details I'm like "oh yeah in this case I see how the cost benefit makes sense". Just more speaking in generalities.