Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why frozen test fixtures are a problem on large projects and how to avoid them

radanskoric.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#51

Post by japhyr »

strehldev wrote: post #46
dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

This is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities. My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address. So it was immedia…

How did you deal with reproducibility when your tests use randomized data? Do you run with a random seed or something, so you can reproduce failures when they come up?

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#52

Post by tclancy »

I feel this is all contentious because, like so much in coding, we have people taking Thing That Works For Me on their current project or in their experience over time and declaring it to be The One True Way while other people are in completely different codebases with different priorities around shipping v quality v cost or what have you and are complaining "That doesn't work for me, Y has always been my go to".

The answer is most definitely, 100%, with no room for argument, to not speak so assuredly, acknowledge other people have the right to think differently and find synthesis and/ or a set of heuristics that apply for given cases.

But this is the Internet, and we need to be arguing PS2 vs X-Box for the rest of our lives, so have at it.

(Me? Factories are great until they aren't, which may not happen if a project or a team is small enough. Generators are great but do have some footguns and I would love to hand over everything to property-based testing, but I _feel_, without any experimenting or trying, they resist anything other than the purest of pure unit tests and can't help with integration tests that much.)

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#53

Post by tclancy »

radanskoric wrote: post #43
dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages). The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite consider…

No, property-based testing is something more like https://hypothesis.readthedocs.io/en/latest/ -- it's like fuzz testing with some smarts and it is lovely where it fits.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#54

Post by FuckButtons »

strehldev wrote: post #46
dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

This is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities. My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address. So it was immedia…

Forgive me if I’m just reading this incorrectly, but that doesn’t sound exactly like property testing as I’ve done it. the libraries implement an algorithm for narrowing down to the simplest reproducer for a given failure mode, so all of the inputs to a test that are randomized are provided by the library.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#55

Post by dkarl »

japhyr wrote: post #51
strehldev wrote: post #46

Earlier quoted context omitted.

This is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities. My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address. So it was immedia…

How did you deal with reproducibility when your tests use randomized data? Do you run with a random seed or something, so you can reproduce failures when they come up?

ScalaCheck includes the random seed in its failure messages, so it's easy to pull the seed out of CI/CD logs and reproduce the failure deterministically.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#56

Post by dkarl »

strehldev wrote: post #46
dkarl wrote: post #36

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead. "Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and c…

This is basically how I solved this in a past codebase. I called them "builders" and for complex scenarios requiring multiple different entities I called them "scenario builders" that created multiple entities. My rule was to randomize every property by default. The test needs to specify which property needs to have a certain value. E.g. set the address if you're testing something about the address. So it was immedia…

You should see if your language has a property-based testing library; it'll have a ton of useful functionality to help with what you're already doing!

A clarification on terminology, the "property" in "property-based testing" refers to properties that code under test is supposed to obey. For example, in the author's Example 2, the collection being sorted is the property that the test is checking.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#58

Post by sceptic123 »

radanskoric wrote: post #6
RHSeeger wrote: post #4

Example 1 bothers me. It says > This test has just made it impossible to introduce another active project without breaking it, even if the scope was not actually broken. Add a new variant of an active project for an unrelated test and now you have to also update this test. And then goes on to test that the known active projects are indeed included in what the call to Project.active returns. However, that doesn't test…

Author here. Thanks for writing up your thoughts on this! The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that: ``` refute_includes active_projects, projects(:inactive) ``` Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ... Regarding the fact that the test does not check that the scope retu…

There's always a scenario where this can break though. What happens if someone introduces a test that confirms that marking `active1` as inactive works. Then it depends on the test order whether your initial test still passes.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#59

Post by radanskoric »

tclancy wrote: post #53
radanskoric wrote: post #43

Earlier quoted context omitted.

Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages). The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite consider…

No, property-based testing is something more like https://hypothesis.readthedocs.io/en/latest/ -- it's like fuzz testing with some smarts and it is lovely where it fits.

Ah, ok, now I understand. Ok, I wasn't talking about that. From what I understand about property based testing it's sort of half way between regular example based testing and formal proofs: It tries to prove a statement but instead of a symbolic proof it does it stohastically via a set of examples?

Unfortunately, I'm not aware of a good property based testing library in Ruby, although it would be useful to have one.

Even so I'm guessing that property based testing in practice would be too resource intensive to test the entire application with it? You'd probably only test critical domain logic components and use regular example tests for the rest.

Re: Why frozen test fixtures are a problem on large projects and how to avoid them

#60

Post by radanskoric »

RHSeeger wrote: post #50
radanskoric wrote: post #33

Earlier quoted context omitted.

I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree: > Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project. Imagine this: 1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write. 2.…

> If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed? No, _but_ (and this is a big _but_) you're not testing the contract of the method, which (presumably) is to return all and only active projects. Testing that it returns _some_ of the active methods is useful, but there are cases where it won't point ou…

I'm with you on the one assertion per test. I bundled two assertions into the same test here because my whole point was to have them effectively together describe a single test, just in a more maintainable manner.

Regarding the fact that I'm not fully testing the contract of the method, you're absolutely correct. But also, no example based test suite is fully doing that. As long as the test suite is example based it is always possible to find a counter-case where the contract is violated but the test suite misses it.

These counter-cases will be more contrived and less likely the better the test suite. So all of us at some point decide that we've done enough and that more contrived cases are so unlikely and the cost of mistake is so small that it's not worth it to put in the extra testing effort. Some people don't explicitly think about it but that decision is still made one way or another.

This is a long way of saying that I both agree with you but that also, in most cases, I would still take the tradeoff and go for more maintainable tests.

Post reply on HN