Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why frozen test fixtures are a problem on large projects and how to avoid them

radanskoric.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#41

Post by bluGill »

orwin wrote: post #31
matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

People really use fixture to simulate internal code? i thought it was overwhelmingly used to simulate external API response, or weird libraries that need some context switching (and in that case, an advice: the NIH syndrome is _very_ valid, and sometime the library you use isn't worth the time you put "fixing" it: just rewrite the damn thing) [edit] though in my case we have one fixture that load a json representatio…

Simulating an external API is the responsibility of a test double of some sort, not a fixture. Fixtures often setup the test doubles with test data, but they are not the test double. Fixtures can setup other things as well (the line between factories and fixtures is blurry)

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#42

Post by yxhuvud »

radanskoric wrote: post #7
jrochkind1 wrote: post #3

factories definitely are the cause of my test suite being slower than I'd like, I've been thinking of switching to fixtures. Good to see some context of what challenges I might be dealing with instead if I do.

Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records. Bt…

What I feel is really missing from factories is the ability to do bulk inserts of a whole chain of entries (including of different kinds). That is where 95% of the inefficiency comes from. As an additional bonus it would make it easy to just list everything single record that was created for a spec

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#43

Post by radanskoric »

dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages).

The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably.

If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#44

Post by bluGill »

disgruntledphd2 wrote: post #32
ozim wrote: post #17

Earlier quoted context omitted.

Inheritance everywhere leads to hell. As much in applications code it is easy to curb, for test code it is just really hard to get people to understand all this duplication that should be there in tests is GOOD.

As always, there's a tradeoff. I used to go for doing all setup in each test for clarity, but one of my co-workers eventually convinced me that doing this in a fixture is better. There'll always be some duplication, but too much makes it harder to see the important stuff in a test.

It depends on how much setup is done, and where it is. 10 tests that share a setup fixture are good. 100,000 starts to get unmaintainable.

I have lots of test fixtures each responsible for about 10 tests. It is very common to have 10-20 tests that share a startup configuration and then adjust it in various ways.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#45

Post by radanskoric »

onionisafruit wrote: post #35
radanskoric wrote: post #26

Earlier quoted context omitted.

I'm familiar with snapshot testing for UI and I agree with you, they can work really well for this because they're usually quick to verify. And especially if you can build in some smart tolerance to the comparison logic, it can be really easy to maintain. But how would you do snapshot testing for behaviour? I'm approaching the problem primarily from the backend side and there most tests are about behaviour.

I'm also primarily on the back end. Like most backenders, I spend my workdays on http endpoints that return json. When I test these the "snapshot" is a json file with a pretty-printed version of the endpoint's response body. Tests fail when the file generated isn't the same as the existing file.

Ah, Ok, yes, for API endpoints it makes a lot of sense. Especially if it's a public API, you need to inspect the output anyway, to ensure that the public contract is not broken.

But, I spend very little or no time on API endpoints since I don't work on projects where the frontend is an SPA. :)

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#46

Post by strehldev »

dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

This is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities.

My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address.

So it was immediately obvious which properties a test relied on.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#47

Post by orwin »

I missed the "frozen". Ok, i understand the text better now. I think the issue is only with the frozen though, i understand why people would think it is necessary, but i think it should be avoided as much as possible, and fixtures, like data should be rewritten each time a data model change.

We have a solution. Not sure if it is elegant, but use it as an inspiration: it works.

When our project run its test, it will generate its database json representation itself (only using its models) with a file that contain fake/test data. That database representation will be loaded in the dev environment, and also in the database fixture that then run our tests. If our tests pass and we have an issue in dev, that mean our test missed something (that happen waaaaaay more often that i like to admit) and we have to add them.

Forcing every test to use this representation also force us to have a dev environment that contain enough items to run the test, and we can't forget to generate an item in the dev database, since that would mean our new feature isn't tested.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#48

Post by stephen »

These two suggestions are fine, but I don't think they make fixtures really that much better--they're still a morass of technical debt & should be avoided at all costs.

The article doesn't mention what I hate most about fixtures: the noise of all the other crap in the fixture that doesn't matter to the current test scenario.

I.e. I want to test "merge these two books" -- great -- but now when stepping through the code, I have 30, 40, 100 other books floating around the code/database b/c "they were added by the fixture" that I need to ignore / step through / etc. Gah.

Factories are the way: https://joist-orm.io/testing/test-factories/

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#49

Post by dkarl »

radanskoric wrote: post #43
dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages). The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite consider…

Oh, that's a very different set of requirements than I was thinking, and I missed that context even though you did mention database testing at one point. You're right, property-based testing is less helpful in that situation, because your database may contain legacy data that your current application code must be able to read but also shouldn't be able to write.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#50

Post by RHSeeger »

radanskoric wrote: post #33
RHSeeger wrote: post #16

Earlier quoted context omitted.

I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those". > The "doesn't include non-active projec…

I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree: > Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. Imagine this: 1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write. 2.…

> If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?

No, _but_ (and this is a big _but_) you're not testing the contract of the method, which (presumably) is to return all and only active projects.

Testing that it returns _some_ of the active methods is useful, but there are cases where it won't point out an issue. For example, image

- Over time, more tests are added "elsewhere" that use the same fixtures

- More active projects are added to the fixture to support those tests

- The implementation in the method is changed to be faster, and an off-by-one error is introduced; so the last project in the list isn't returned

In that ^ case, testing that _some_ of the active projects are returned will still return true; the bug won't be noticed.

Not directly related to the above, but I'll note that I would also split 2/3 into different tests.

- Make sure all projects returned are active

- Make sure projects returned includes all active projects

I think that's more of a style thing, but I _try_ to stick to each test testing one and only one thing. I don't always do that, but it's a rule of thumb for me.

Post reply on HN