Composable Tests

> All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.

What? By design that's exactly how virtually all test frameworks are designed to work.

> It's the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.

Nonsense. I've been writing unit tests for years in multiple programming languages and using multiple frameworks, and not once did I stumbled upon any scenario whatsoever where I had to be mindful about isolating tests. Because that was assured and a given from the very start.

I wonder what kind of convoluted mess you are doing with your tests.

If these tests are being run just once, the cost of creating them will likely vastly outweigh the cost of running them.

If they are run more than once, they can be reordered randomly each time at zero marginal cost.

I disagree so hard with the notion that isolation is somehow done for you, that I don't even know how to begin arguing.

I'll admit to starting this confusion by reading what I thought would be written, not what was actually there. TFA does phrase it to my satisfaction that the test is isolated. I assumed when reading it that Kent thought he had isolation but did not. By analogies: a naive programmer does not perceive the existence of race conditions; a naive web app developer doesn't perceive the existence of CAP; a naive backend engineer thinks they have full ACID guarantees when they're only guaranteed READ_COMMITTED. I thought I was reading another writer just wishing isolation into existence through ignorance.

(So I accidentally straw-manned Kent's argument, but maybe that's not such a problem, because it seems you're OK with the straw man version?)

Anyway, maybe a question moves things forward.

JUnit's been around for decades. Every developer has had multicore dev machines for decades. Why would parallel execution not be the default, or a simple toggle that everyone would turn on immediately?

This is not at all how it works.

But you missed the important distinction: random is bad because it repeats unnecessarily. One of the typically used examples in statistics 101 class to drive attention to this fact is the question posed to students to guess, without actually doing any math: what is the chance of two people in a room with (however many students are there in the class) to have a birthday on the same day?

People consistently estimate the chance significantly lower than what it actually is. And this error is supposed to encourage students to dedicate more attention to the subject because they will have found their intuition to be so woefully inadequate.

See this for the exact math: https://en.wikipedia.org/wiki/Birthday_problem .

* * *

Now, if you are interested in economics of writing tests and running them (but you don't have to be, unless it's your job):

First of all, for this problem, it doesn't matter how much it costs to write the tests. We are talking about the strategy to cover more feature combinations in the test, so, no matter how much it costs to write a test, the cost (per test) will be the same regardless of the strategy chosen. The combination is where the difference will manifest itself.

Second, tests are normally written by salaried employees. This makes the cost of writing an individual test very difficult to divine (and even if you could, it would be a useless metric). It's difficult to calculate because a lot of factors that contribute to writing a test happen in aggregates (eg. the tester needs to explore and familiarize themselves with the system before writing a test, but this effort contributes to multiple tests). Salaried employees are paid a fixed amount regardless of the intensity of work: if they produce one, ten or a hundred tests per month, the salary is the same. Tests can be vastly different in scope and in an effort necessary to implement them, however the effort necessary to implement a test isn't indicative of the value of the test (it's possible that a very simple test will be very effective at detecting problems where a very complex test isn't and the other way around).

All this said, suppose you did come up with a way of calculating the cost of writing an individual test: it still doesn't help you to estimate the cost of testing the product, it doesn't even move the needle on this question. It changes nothing about your QA department budget. It changes nothing about delivery schedule. It's just a number you can marvel at in your spare time...