Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why frozen test fixtures are a problem on large projects and how to avoid them

radanskoric.com

Post reply on HN
Print view View original

11–20 of 67 posts

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#11

Post by onionisafruit »

I’ve found that golden master tests (aka snapshot testing) pair very well with fixtures. If I need to add to the fixtures for a new test, I regenerate the golden files for all the known good tests. I barely need to glance at these changes because, as I said, they are known good. Still I usually give them a brief once over to make sure I didn’t do something like add too many records to a response that’s supposed to be a partial page. Then I go about writing the new test and implementing the change I’m testing. After implementing the change, only the new test’s golden files should change.

They are also nice because I don’t have to think so much about assertions. They automatically assert the response is exactly the same as before.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#12

Post by swader999 »

I find inheritance in tests leads quickly to hell. Striving for every last bit of reuse seems like the right thing to do but it hurts in subtle ways that compound over time. If you must, use composition and spend the time on a DSL that clearly documents the setup in each test.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#13

Post by matsemann »

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database.

Instead, if you're able to decouple the ORM from your application, with a separate layer, and instead pass plain objects around (not fat db backed models), one is much freer to write code that's "pure". This input gives that output. For tests like these one only needs to create whatever data structure the function desires, and then verify the output. Worst case verify that it called some mocks with x,y,z.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#14

Post by axelthegerman »

matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

Right let's add more layers that always solves everything.

No language or abstraction is perfect but if someone prefers pure functional coding, Rails and Django are just not it, don't try to make them. Others like em just as they are

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#15

Post by erdaniels »

I think fixtures generally work fine. If a change to one breaks many tests, introduce a new one and start using that. I also think it's okay to make some manual changes to them in the test and it's distinct from wanting factories; needing factories only in test code feels like a waste.

100% agree with "Test only what you want to test".

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#16

Post by RHSeeger »

radanskoric wrote: post #6
RHSeeger wrote: post #4

Example 1 bothers me. It says > This test has just made it impossible to introduce another active project without breaking it, even if the scope was not actually broken. Add a new variant of an active project for an unrelated test and now you have to also update this test. And then goes on to test that the known active projects are indeed included in what the call to Project.active returns. However, that doesn't test…

Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope retu…

I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those".

> The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that:

You're correct; I totally missed that.

> In either case it is up to you to make sure that the projects are representative.

That's fair, but that's also the point you're trying to address / make more robust by how you're trying to write tests (what the article is about). Specifically

- The article is about: How to make sure you're tests are robust against test fixtures changing

- That comment says: It's up to you to make sure your test fixtures don't change in a way that breaks your tests

> You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain.

By restricting your fixtures to just the projects (that are relevant to the test), you're making _the tests_ easier to maintain; not just the one test but the test harness as a whole. What I mean is that you're reducing "action at a distance". When you modify the data for your test, you don't need to worry about what other tests, somewhere else, might also be impacted.

Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.

All that being said, what I'm talking about relies on creating the test data local to the tests. And doing that has a cost (time, generally). So there's a tradeoff there.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#17

Post by ozim »

swader999 wrote: post #12

I find inheritance in tests leads quickly to hell. Striving for every last bit of reuse seems like the right thing to do but it hurts in subtle ways that compound over time. If you must, use composition and spend the time on a DSL that clearly documents the setup in each test.

Inheritance everywhere leads to hell.

As much in applications code it is easy to curb, for test code it is just really hard to get people to understand all this duplication that should be there in tests is GOOD.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#18

Post by mnutt »

jrochkind1 wrote: post #3

factories definitely are the cause of my test suite being slower than I'd like, I've been thinking of switching to fixtures. Good to see some context of what challenges I might be dealing with instead if I do.

I have a large rails app that was plagued with slow specs using factory_bot. Associations in factories are especially dangerous given how easy it is to build up big dependency chains. The single largest speedup was noting that nearly every test was in the context of a user and org, and creating a default_user and default_org fixture.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#19

Post by radanskoric »

jrochkind1 wrote: post #9
radanskoric wrote: post #7

Earlier quoted context omitted.

Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records. Bt…

Thanks! While I have you, since you seem to know what's up with this stuff, I'm going to ask you a question I have been curious about, in Rails land too. While I see the pro's (and con's) of fixtures, one thing I do _not_ like is Rails ordinary way of specifying fixtures, in yaml files. Especially gets terrible for associations. It's occured to me there's no reason I can't use FactoryBot to create what are actually f…

Your thinking is sound. At the end of the day Rails default fixtures is nothing more than some code that reads yaml files and creates records once at the start of test suite run.

So you can definitely use FactoryBot to create them. However, the reason I think that's rarely done is that you're pretty likely to start recreating a lot of the features of Rails fixtures yourself. And perhaps all you need to do is to dynamically generate the yaml files. Rails yaml fixtures are actually ERB files and you can treat is an ERB template and generate its code dynamically: https://guides.rubyonrails.org/testing.html#embedding-code-i...

If that is flexible enough for you, it's a better path since you'll get all the usual fixture helpers and association resolving logic for free.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#20

Post by antonymoose »

axelthegerman wrote: post #14
matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

Right let's add more layers that always solves everything. No language or abstraction is perfect but if someone prefers pure functional coding, Rails and Django are just not it, don't try to make them. Others like em just as they are

Three clean and simple layers that dovetail are better than one ball-of-yarn God class with too many dependencies.
Post reply on HN