Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

141–150 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#141

Post by eru »

thunky wrote: post #85
eru wrote: post #50

Earlier quoted context omitted.

Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default. But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem. Also, even with Hypothesis you can and…

> But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly? Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in p…

You can think of property based tests as defining a vast 'universe' of tests. Like 'for all strings S and T, we should have to_upper(S) + to_upper(T) == to_upper(S+T)' or something like that. That defines an infinite set of individual test cases: each choice for S and T gives you a different test case.

Running all of these tests would take too long. So instead we take a finite sample from our universe, and only run these.

> With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.

You could always take the same sample, of course. But that means you only ever explore a very small fraction of that universe. So it's more likely you miss something. Remember: closing your eyes doesn't make the tiger go away.

If there are important cases you want to have checked every time, you can use the @example decorator in Hypothesis, or you can just write a traditional example based test.

Re: Hypothesis: Property-Based Testing for Python

#142

Post by eru »

nyrikki wrote: post #102
thunky wrote: post #85

Earlier quoted context omitted.

> But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly? Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in p…

Saving the seed in the build artifacts/logs has saved a lot of time for me even with tools like faker.

Yes, you should note the seed you use for a run in the logs, but you should use a new seed each run (unless trying to reproduce a known bug) so you cover more of the search space.

Re: Hypothesis: Property-Based Testing for Python

#143

Post by eru »

rmunn wrote: post #135
dkarl wrote: post #134

Earlier quoted context omitted.

That's pretty cool, and now I'm curious if there's something similar for ScalaCheck. My comment comes from my own experience, though, introducing Hypothesis and ScalaCheck into codebases and quickly causing noticeable increases in unit test times. I think the additional runtime for tests is undoubtedly worth it, but maybe not a good trade-off when people are used to running unit tests several times an hour as part of…

Or a watch command that runs tests in the background on save, and an IDE setting to flag code when the watched tests produce a failure. Get used to that, and it's not even a matter of stopping to run the tests: they run every time you hit Ctrl-S, and you just keep on typing — and every so often the IDE notifies you of a failed test. The drawback is that you might get used to saying "Well, of course the tests failed,…

You can also set things up so that you only run 10 examples per test when doing a quick check during development, but your CI runs the full 200 examples per test (or even more).

Re: Hypothesis: Property-Based Testing for Python

#144

Post by eru »

bunderbunder wrote: post #99
rmunn wrote: post #7

I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (…

I love them for this, too. Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent. (I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a librar…

You could just hard code a PRNG seed, and then the test would trivially be deterministic? I'm not sure what they say their objection is, is actually their real objection?

Re: Hypothesis: Property-Based Testing for Python

#145

Post by eru »

dd82 wrote: post #109
NortySpock wrote: post #2

I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "sever…

Essentially this is a good example of parametrized tests, just supercharged with generated inputs. So if you already have parametrized tests, you're already halfway there.

Yes, when I saw eg Golang people use table driven tests like this, I was wondering why nobody seems to have told them about generating these tables automatically..

Re: Hypothesis: Property-Based Testing for Python

#146

Post by chriswarbo »

lgas wrote: post #73
eru wrote: post #20

Earlier quoted context omitted.

And Hypothesis is miles ahead of QuickCheck in how it handles shrinking! Not only does it shrink automatically, it has no problem preserving invariants from generation in your shrinking; like only prime numbers or only strings that begin with a vowel etc.

QuickCheck also shrinks automatically and preserves invariants though?

QuickCheck won't preserve invariants, since its shrinkers are separate from its generators. For example:

    data Rat = Rat Int Nat deriving (Eq, Show)

    genRat = do
      (num, den)
`genRat` is a QuickCheck generator. It cannot do shrinking, because that's a completely separate thing in QuickCheck.

We can write a shrinker for `Rat`, but it will have nothing to do with our generator, e.g.

    shrinkRat (Rat num den) = do
      (num', den')
Sure, we can stick these in an `Arbitrary` instance, but they're still independent values. The generation process is essentially state-passing with a random number generator; it has nothing to do with the shrinking process, which is a form of search without backtracking.

    instance Arbitrary Rat where
      arbitrary = genRat
      shrink = shrinkRat
In particular, `genRat` satisfies the invariant that values will have non-zero denominator; whereas `shrinkRat` does not satisfy that invariant (since it shrinks the denominator as an ordinary `Nat`, which could give 0). In fact, we can't even think about QuickCheck's generators and shrinkers as different interpretations of the same syntax. For example, here's a shrinker that follows the syntax of `genRat` more closely:

    shrinkRat2 (Rat n d) = do
      (num, den)
This does have the invariant that its output have non-zero denominators; however, it will get stuck in an infinite loop! That's because the incoming `d` will be non-zero, so when `shrink` tries to shrink `(n, d)`, one of the outputs it tries will be `(n, 0)`; that will lead to `Rat n 1`, which will also shrink to `Rat n 1`, and so on.

In contrast, in Hypothesis, Hedgehog, falsify, etc. a "generator" is just a parser from numbers to values; and shrinking is applied to those numbers, not to the output of a generator. Not only does this not require separate shrinkers, but it also guarantees that the generator's invariants hold for all of the shrunken values; since those shrunken values have also been outputted by the generator (when it was given smaller inputs).

Re: Hypothesis: Property-Based Testing for Python

#147

Post by eru »

NortySpock wrote: post #116
fwip wrote: post #3

Earlier quoted context omitted.

I think the easiest way is to start with general properties and general input, and tighten them up as needed. The property might just be "doesn't throw an exception", in some cases. If you find yourself writing several edge cases manually with a common test logic, I think the @example decorator in Hypothesis is a quick way to do that: https://hypothesis.readthedocs.io/en/latest/reference/api.ht...

Thanks, the "does not throw an exception" property got my mental gears turning in terms of how to get started on this, and from there I can see how one could add a few more properties as one goes along. Appreciate you taking the time to answer.

Yes. One of the first things you have to do when writing any property based tests, no matter what you actually want to test, is defining your input generators. And once you have those generators, you might as well throw together the "doesn't crash unexpectedly" test together.

That not only tests your code, but also exercises your just written input generators.

Re: Hypothesis: Property-Based Testing for Python

#148

Post by eru »

masklinn wrote: post #68
benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

What you're talking about is using an oracle (a different implementation of what you're testing), it's an option for property (or exhaustive) testing but is by no means a requirement. Even for a sort function there are plenty of properties you can check without needing an oracle e.g. - that the output sequence is the same length as the input - that the output sequence is sorted - that the population counts are the sa…

> Having a generalised sort doesn't mean you can't write a more specialised one which better fits your data set e.g. you might be in a situation where a radix sort is more appropriate.

The opposite might also be true. Suppose you already have a specialised implementation, and you want to write a new generalised one. You can still test them against each other.

Eg suppose you are writing a library that supports sorting crazy huge datasets that don't fit into memory. You can still check that it gives the same answers as the built-in sorting algorithm from the standard library on input that's small enough to fit into memory.

Re: Hypothesis: Property-Based Testing for Python

#149

Post by chriswarbo »

teiferer wrote: post #79
eru wrote: post #24

Earlier quoted context omitted.

> 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert No. That's one valid approach, especially if you have a simpler alternative implementation. But testing against an oracle is far from the only property you can check. For your example: suppose you have implemented an add function for your fancy new data type (perhaps it's a crazy vector/tensor th…

> a + b == b + a > a + (b + c) = (a + b) + c > a + (-a) == 0 Great! Now I have a stupid bug that always returns 0, so these all pass, and since I didn't think about this case (otherwise I'd not have written that stupid bug in the first place), I didn't add a property about a + b only being 0 if a == -b and boom, test is happy, and there is nothing that the framework can do about it. Coming up with those properties is…

> Just doing this as an afterthought by playing lottery and trying to come up with smart properties after the fact is not going to get you the best outcome.

This sounds backwards to me. How could you write any tests, or indeed implement any functions, if you don't know any relationships between the arguments/return-value, or the state before/after, or how it relates to other functions, etc.?

For the addition example, let's say the implementation includes a line like `if (x == 0) return y`; would you seriously suggest that somebody writing that line doesn't know a property like `0 + a == a`? Would that only be "an afterthought" when "trying to come up with smart properties after the fact"? On the contrary, I would say that property came first, and steered how the code was written.

Incidentally, that property would also catch your "always returns 0" counterexample.

I also don't buy your distinction that "real life code" makes things much harder. For example, here's another simple property:

    delete(key); assert lookup(key) == []
This is pretty similar to the `a + (-a) == 0` example, but it applies to basically any database; from in-memory assoc-lists and HashMaps, all the way to high-performance, massively-engineered CloudScale™ distributed systems. Again, I struggle to imagine anybody implementing a `delete` function who doesn't know this property; indeed, I would say that's what deletion means. It's backwards to characterise such things as "com[ing] up with smart properties after the fact".

Re: Hypothesis: Property-Based Testing for Python

#150

Post by Jtsummers »

imiric wrote: post #130
Jtsummers wrote: post #121

Earlier quoted context omitted.

My experience is that PBT tests are mostly hard in devising the generators, not in the testing itself. Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT, here are some of the tests as both PBT-style and example-based style: @given(st.integers()) def test_left_identity_pbt(a): assert add(a, 0) == a def test_left_identity(): assert add(10…

> Since it came up in another thread (yes, it's trivial), a function `add` is no easier or harder to test with examples than with PBT Come on, that example is practically useless for comparing both approaches. Take a look at the article linked above. The amount of non-trivial code required to setup a PBT should raise an eyebrow, at the very least. It's quite possible that the value of such a test outweighs the comple…

> Come on, that example is practically useless for comparing both approaches.

Come on, I admitted it was trivial. It was a quick example that fit into a comment block. Did you expect a dissertation?

> that implementing all the test variations with EBT would be infeasible

That's kind of the point to my previous comment. PBTs will generate many more examples than you would create by hand. If you have EBTs already, you're one step away from PBTs (the generators, I never said this was trivial just to preempt another annoying "Come on"). And then you'll have more comprehensive testing than you would have had sticking to just your carefully handcrafted examples. This isn't the end of property-based testing, but it's a really good start and the easiest way to bring it into an existing project because you can mostly reuse the existing tests.

Extending this, once you get used to it, to stateful testing (which many PBT libraries support, including Hypothesis) you can generate a lot of very useful end-to-end tests that would be even harder to come up with by hand. And again, if you have any example-based end-to-end tests or integration tests, you need to come up with generators and you can start converting them into property-based tests.

> but choosing one strategy over the other should be a conscious decision made by the team.

Ok. What prompted this? I never said otherwise. It's also not an either/or situation, which you seem to want to make it. As I wrote in that previous comment, you can use both and use the property-based tests to bolster the example-based tests, and convert counterexamples into more example-based tests for your regression suite.

> I haven't actually written PBTs myself.

Huh.

Post reply on HN