Why frozen test fixtures are a problem on large projects and how to avoid them

In theory you're 100% right, a true unit test is completely isolated from the rest of the system and than a lot of the problems disappear.

In reality, that is also not free. It imposes some restrictions on the code. Sometimes being pragmatic, backing off from the ideal leads to faster development and quicker deliver of value to the users. Rails is big on these pragmatic tradeoffs. The important thing is that we know when and why we're making the tradeoff.

Usually I go with Rails defaults and usually it's not a problem. Sometimes, when the code is especially complex and perhaps on the critical path, I turn up the purity dial and go down the road you describe exactly for the benefits you describe.

But when I decide that sticking to the defaults the is right tradeoff I want to get the most of it and use Fixtures (or Factories) in the optimal way.

When you say inheritance do you mean DRY as in "Don't repeat yourself"?

I'm not sure what you mean by inheritance in tests but DRY is criminally overused in tests. That could be a whole separate article but the tradeoffs are very different between test and app code and repetition in the test code is much less problematic and sometimes even desirable.

I've been doing one thing for many years that forces me to be smart about test fixtures in my integration tests and test data: I test concurrently with many threads.

This means that my test can't depend on the database to be in some known state or assume to have exclusive access to that database. And for example modify anything that might be used by another test. They can only modify things that are specific to that test.

Most of my tests work around this limitation by either just creating their own teams, users, and other objects they need with randomized ids; or in some cases deferring their execution until some bit of logic with lock has created some shared data that then is never modified.

Instead of hard coded IDs, I tend to use randomized ids (UUIDs typically). I have a person data generator that gives me human readable names, email addresses, etc. Randomized data like this avoids test modifying each other's data.

As an example, we have a few tests for an analytics dashboard that locks on a bit of expensive code that creates a lot of content via our APIs to do analytics on. The scenario is quite elaborate and uses a few factories, known timestamps, etc. If I refactor my data model, my factories are also refactored. Using a lock ensures that data is initialized only once. Once that is done, there are a bunch of test that that test different queries against that.

You might think that all this is slow. It's not. I have about 380 integration tests like this that run in under 30 seconds on my laptop (which has a lot of CPU cores). Having this as a safety net is very empowering. I've been on teams that had less tests where running them took ten or more minutes. This I can do quickly before committing.

Testing like this has many advantages. But one includes easy to maintain tests. I put some effort into usable test data factories. The "when" part of a BDD style integration tests is usually most of the work. So, by making that as easy as I can, I lower the barrier for writing more tests. And using all my cpu cores, minimizes the impact new tests have on execution time to the point where I don't worry about that.

Another is that for big structural changes my tests continue to work if I just fix their shared factories to do the right thing usually.

there's a profiler that can show you what to focus on, probably fprof here: https://test-prof.evilmartians.io/ (been a while and I don't remember exactly what I used)

(now maybe that's what you used to see what was causing the slowdown, but mentioning to for others to help them identify the bottlenecks.)

I disagree. Prefer integration tests to unit tests where ever possible. If your tests run fast - which most integration tests should be able to do; and you are running your tests often - there is no downside. Your tests run fast and since you run them often you always know what broke: the last thing you changed.

Fixtures done right ensure that everyone starts with a good standard setup. The question is WHAT state the fixture setups. I have a fixture that setups a temporary data directory with nothing in it - you can setup your state, but everything will read from that temporary data directory.

Unit tests do have a place, but most of us are not writing code that has a strong well defined interface that we can't change. As such they don't add much value since changes to the code also imply changes to the code that uses them. When some algorithm is used in a lot of places unit tests it well - you wouldn't dare change it anyway, but when the algorithm is specific to the one place that calls it then there is no point in a separate test for it even though you could. (there is a lot of grey area in the middle where you may do a few unit tests but trust the comprehensive integration tests)

> Worst case verify that it called some mocks with x,y,z.

That is the worst case to avoid if at all possible (sometimes it isn't) that a function is called is an implementation details. Nobody cares. I've seen too many tests fail because I decided to change a function signature and now there is a new parameter A that every test needs to be updated to expect. Sometimes this is your only choice, but mock heavy tests are a smell in general and that is really what I'm against. Don't test implementation details, test what the customers care about is my point, and everything else follows from that (and where you have a different way that follows from that it may be a good think I want to know about!)