Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why frozen test fixtures are a problem on large projects and how to avoid them

radanskoric.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#61

Post by radanskoric »

sceptic123 wrote: post #58
radanskoric wrote: post #6

Earlier quoted context omitted.

Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope retu…

There's always a scenario where this can break though. What happens if someone introduces a test that confirms that marking `active1` as inactive works. Then it depends on the test order whether your initial test still passes.

It's required for tests to clean up after themselves. With Rails and fixtures this is handled by default: each test runs inside a transaction which is rolled back at the end of the test. That way each test starts with the same initial state.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#62

Post by radanskoric »

stephen wrote: post #48

These two suggestions are fine, but I don't think they make fixtures really that much better--they're still a morass of technical debt & should be avoided at all costs. The article doesn't mention what I hate most about fixtures: the noise of all the other crap in the fixture that doesn't matter to the current test scenario. I.e. I want to test "merge these two books" -- great -- but now when stepping through the cod…

Author here. I didn't mention it because I wasn't writing an evaluation of fixtures. Just writing about how to make better use of fixtures. I actually use both fixtures and factories depending on the project specifics and also whether it is even my decision to make. :)

Personally, I even slightly prefer to use Factories and I also previously wrote about a better way to use them: https://radanskoric.com/articles/test-factories-principal-of...

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#63

Post by swader999 »

radanskoric wrote: post #23
swader999 wrote: post #12

I find inheritance in tests leads quickly to hell. Striving for every last bit of reuse seems like the right thing to do but it hurts in subtle ways that compound over time. If you must, use composition and spend the time on a DSL that clearly documents the setup in each test.

When you say inheritance do you mean DRY as in "Don't repeat yourself"? I'm not sure what you mean by inheritance in tests but DRY is criminally overused in tests. That could be a whole separate article but the tradeoffs are very different between test and app code and repetition in the test code is much less problematic and sometimes even desirable.

Both actually. But having to open up three files to figure out how this thing is setup and then override setup to change it slightly in my one case. You get the idea. A really good DSL can help in the areas where creating the SUT is very complex.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#64

Post by ozim »

disgruntledphd2 wrote: post #32
ozim wrote: post #17

Earlier quoted context omitted.

Inheritance everywhere leads to hell. As much in applications code it is easy to curb, for test code it is just really hard to get people to understand all this duplication that should be there in tests is GOOD.

As always, there's a tradeoff. I used to go for doing all setup in each test for clarity, but one of my co-workers eventually convinced me that doing this in a fixture is better. There'll always be some duplication, but too much makes it harder to see the important stuff in a test.

I guess one level of inheritance is bearable, downside is once you start, there will be people coming in later adding more.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#65

Post by matsemann »

onionisafruit wrote: post #28
matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

Fixtures are great for integration tests. But I agree that unit tests needing fixtures indicates a design issue. Still, most of us work on code bases with design issues either of our own making or somebody else’s.

Yup, so I'm not against fixtures per se, they have their uses and can be a pragmatic choice. I just often don't like when I have to use them, as it's often to patch over something else. But things are never perfect.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#66

Post by matsemann »

bluGill wrote: post #29
matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

I disagree. Prefer integration tests to unit tests where ever possible. If your tests run fast - which most integration tests should be able to do; and you are running your tests often - there is no downside. Your tests run fast and since you run them often you always know what broke: the last thing you changed. Fixtures done right ensure that everyone starts with a good standard setup. The question is WHAT state the…

I guess it depends a bit on what you work on. Lately I'm working on algorithm heavy stuff, then testing input=>output is much more valuable than if things run, easier to consider edge cases when calling it directly etc. But if you make a crud app it's often more useful to test a flow. So depends.

As for mocks I don't disagree, hence calling it worst case.

What often works for me is separating the code. For instance if I call a function that first queries the db and then marshall that data into something, it's often easier to test it by splitting it. One function that queries, that one can test with some db fixtures or other setup. And then another that gets a model in and only does the pure logic and returns the result. Can then be tested separately. And then a third function which is the new one, that just calls the first and pass the result into the second. Can be boilerplaty, so again, depends.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#67

Post by immibis »

bluGill wrote: post #40
immibis wrote: post #10

Your test fixtures are introducing tighter coupling between the tests, than the code they are testing! In this scenario (a mock DB with data that a test relies on, which is incompatible with a new test you want to add), duplicating the fixture is correct. Different tests with incompatible requirements on state should use different mock data. In this specific case, however, it's also true that one of them can be modif…

That depends on what state the fixture sets up and when you run them. That state becomes something you expect all tests to handle, if you change the fixture state and test breaks you should be fixing the production code - not the test code - to handle that change. Of course in reality I can well believe it is a tests data conflict in most cases (someone with the name "test user one" already is in the database at a di…

Different unit tests work on different state. One unit test says that if I start from a blank database, add two users, now there are two users. A different unit test says that if I start from a database with two users, and list the users, I get them.

If you rely on the add-user test running before the list-users test, you introduce bad coupling between tests. However, you could run them all in order in one test - it would be a more complex test. You could then run that on a real database and call it an integration test.

Tests at all levels of complexity are useful. You could run end-to-end tests on a production e-commerce system, from signup to payment with a real credit card and delivery of a real physical product, if you wanted. Backup power systems are tested by shutting off the power a few times a year.

But this article is clearly about unit tests. You shouldn't run unit tests against the production database and you should aim to minimize their dependencies.

Post reply on HN