Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why frozen test fixtures are a problem on large projects and how to avoid them

radanskoric.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#31

Post by orwin »

matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

People really use fixture to simulate internal code? i thought it was overwhelmingly used to simulate external API response, or weird libraries that need some context switching (and in that case, an advice: the NIH syndrome is _very_ valid, and sometime the library you use isn't worth the time you put "fixing" it: just rewrite the damn thing)

[edit] though in my case we have one fixture that load a json representation of our dev dynamodb into moto, and thus we mock internal data, but this data is still read through our data models, it doesn't really replace internal code, only internal "mechanics"

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#32

Post by disgruntledphd2 »

ozim wrote: post #17
swader999 wrote: post #12

I find inheritance in tests leads quickly to hell. Striving for every last bit of reuse seems like the right thing to do but it hurts in subtle ways that compound over time. If you must, use composition and spend the time on a DSL that clearly documents the setup in each test.

Inheritance everywhere leads to hell. As much in applications code it is easy to curb, for test code it is just really hard to get people to understand all this duplication that should be there in tests is GOOD.

As always, there's a tradeoff. I used to go for doing all setup in each test for clarity, but one of my co-workers eventually convinced me that doing this in a fixture is better.

There'll always be some duplication, but too much makes it harder to see the important stuff in a test.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#33

Post by radanskoric »

RHSeeger wrote: post #16
radanskoric wrote: post #6

Earlier quoted context omitted.

Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope retu…

I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those". > The "doesn't include non-active projec…

I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree:

> Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.

Imagine this:

1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write.

2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project".

3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects.

Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power.

If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#34

Post by jrochkind1 »

radanskoric wrote: post #19
jrochkind1 wrote: post #9

Earlier quoted context omitted.

Thanks! While I have you, since you seem to know what's up with this stuff, I'm going to ask you a question I have been curious about, in Rails land too. While I see the pro's (and con's) of fixtures, one thing I do _not_ like is Rails ordinary way of specifying fixtures, in yaml files. Especially gets terrible for associations. It's occured to me there's no reason I can't use FactoryBot to create what are actually f…

Your thinking is sound. At the end of the day Rails default fixtures is nothing more than some code that reads yaml files and creates records once at the start of test suite run. So you can definitely use FactoryBot to create them. However, the reason I think that's rarely done is that you're pretty likely to start recreating a lot of the features of Rails fixtures yourself. And perhaps all you need to do is to dynam…

Cool, thanks!

I feel like i don't _want_ the association resolving logic really, that's what I don't like! And if it's live ruby instead of YAML, it's easy to refer to another fixture object by just looking it up as a fixture like normal? (I guess there's order of operation issues though,hm).

And the rest seems straightforward enough, and better to avoid that "compile to yaml" stage for debugging and such.

We'll see, maybe I'll get around to trying it at some point, and release a perversely named factory_bot_fixtures gem. :)

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#35

Post by onionisafruit »

radanskoric wrote: post #26
onionisafruit wrote: post #11

I’ve found that golden master tests (aka snapshot testing) pair very well with fixtures. If I need to add to the fixtures for a new test, I regenerate the golden files for all the known good tests. I barely need to glance at these changes because, as I said, they are known good. Still I usually give them a brief once over to make sure I didn’t do something like add too many records to a response that’s supposed to be…

I'm familiar with snapshot testing for UI and I agree with you, they can work really well for this because they're usually quick to verify. And especially if you can build in some smart tolerance to the comparison logic, it can be really easy to maintain. But how would you do snapshot testing for behaviour? I'm approaching the problem primarily from the backend side and there most tests are about behaviour.

I'm also primarily on the back end. Like most backenders, I spend my workdays on http endpoints that return json. When I test these the "snapshot" is a json file with a pretty-printed version of the endpoint's response body. Tests fail when the file generated isn't the same as the existing file.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#36

Post by dkarl »

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead.

"Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and can be combined to create generators of other types. (The terminology varies from one library to another. Different libraries use the terms "generators," "arbitraries," and "strategies" in slightly different and overlapping ways.)

For example, if you have a generator for strings and a generator for non-negative integers, it's trivial to create a generator for a type Person(name, age).

Generators can also be filtered. For example, if you have a generator for Account instances, and you need active Account instances in your test, you can apply a filter to the base generator to select only the instances where _.isActive is true.

Once you have a base generator for each type you need in your tests, the individual tests become clear and succinct. There is a learning curve for working with generators, but as a rule, the test code is very easy to read, even if it's tricky to write at first.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#37

Post by perlgeek »

For a database-driven application with sqlalchemy, I've found mixer[0] to be pretty helpful. It gives you an easy way to generate an object, and it automatically creates dummy-objects that your object depends on.

You can also supply defaults and name schemes for individual columns.

For business logic, I prefer to have it structured in a way that it doesn't need the database for testing, but loading and searching stuff from the DB also needs to be tested, and for those, mixer strikes a really good balance. You only need to specify the attributes that are relevant for the test, and you don't need shared fixtures between many tests.

[0]: https://pypi.org/project/mixer/

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#38

Post by jrochkind1 »

mnutt wrote: post #18
jrochkind1 wrote: post #3

factories definitely are the cause of my test suite being slower than I'd like, I've been thinking of switching to fixtures. Good to see some context of what challenges I might be dealing with instead if I do.

I have a large rails app that was plagued with slow specs using factory_bot. Associations in factories are especially dangerous given how easy it is to build up big dependency chains. The single largest speedup was noting that nearly every test was in the context of a user and org, and creating a default_user and default_org fixture.

That's a great, example, thanks.

Then you just refer to the fixture in your factory definitions? Seems very reasonable.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#39

Post by abhashanand1501 »

matsemann wrote: post #13

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database. Instead, if you're ab…

You should look into factory boy (in django). Been using it for 10 years. It helps with this situation.

Foofactory() will automatically setup all the foreign key dependencies.

It can also generate fuzzy data, although having fuzzy data has its own issues in terms of brittle tests (if not done correctly).

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#40

Post by bluGill »

immibis wrote: post #10

Your test fixtures are introducing tighter coupling between the tests, than the code they are testing! In this scenario (a mock DB with data that a test relies on, which is incompatible with a new test you want to add), duplicating the fixture is correct. Different tests with incompatible requirements on state should use different mock data. In this specific case, however, it's also true that one of them can be modif…

That depends on what state the fixture sets up and when you run them. That state becomes something you expect all tests to handle, if you change the fixture state and test breaks you should be fixing the production code - not the test code - to handle that change. Of course in reality I can well believe it is a tests data conflict in most cases (someone with the name "test user one" already is in the database at a different address..) and this is something you need to ensure doesn't happen.

I have a fixture that sets our database to the initial install state. This works for me because we are an embedded system where every month we ship a bunch more new systems and so code needs to see that initial install state, if we change the initial state (which we do all the time) and a test breaks we want to know and fix that since customers will see that situation.

However if you run on a server in a data center I could well believe you will never again see any specific state and so a fixture probably isn't right. Maybe ideally every test would take a snapshot of your current production database and test against that (with whatever additional data you add for the test) - if a customer enters data that breaks a test that is a "all hands on deck" to fix the code before customers hit that code path. Maybe - I don't work in this space and so I'm just speculating what you need.

Post reply on HN