Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

71–80 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#71

Post by lgas »

wongarsu wrote: post #66
benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

Imho it's just not a great example. A better example would be testing the invariants of the function. "For any list of numbers, my_sort does not throw an exception" is trivial to test. "For any list of numbers, in the list returned by my_sort the item at index n is smaller than the item at index n+1" would be another test. Those two probably capture the full specification of my_sort, and you don't need a sort functio…

Always returning the empty list meets your spec.

Re: Hypothesis: Property-Based Testing for Python

#73

Post by lgas »

eru wrote: post #20
Jtsummers wrote: post #12

Earlier quoted context omitted.

> because there is no counterexample shrinking Hypothesis does shrink the examples, though.

And Hypothesis is miles ahead of QuickCheck in how it handles shrinking! Not only does it shrink automatically, it has no problem preserving invariants from generation in your shrinking; like only prime numbers or only strings that begin with a vowel etc.

QuickCheck also shrinks automatically and preserves invariants though?

Re: Hypothesis: Property-Based Testing for Python

#74

Post by wongarsu »

lgas wrote: post #71
wongarsu wrote: post #66

Earlier quoted context omitted.

Imho it's just not a great example. A better example would be testing the invariants of the function. "For any list of numbers, my_sort does not throw an exception" is trivial to test. "For any list of numbers, in the list returned by my_sort the item at index n is smaller than the item at index n+1" would be another test. Those two probably capture the full specification of my_sort, and you don't need a sort functio…

Always returning the empty list meets your spec.

Good point. I suppose we should add "number of input elements equals number of output elements" and "every input element is present in the output". Translated in a straightforward test that still allows my_sort([1,1,2]) to return [1,2,2], but we have to draw the line somewhere

Re: Hypothesis: Property-Based Testing for Python

#75

Post by [deleted] »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

[deleted]

Re: Hypothesis: Property-Based Testing for Python

#76

Post by sevensor »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

I use hypothesis to test that the happy path is really happy. I use it to define the set of expected inputs for a function, and it looks for the nastiest possible sets of inputs. It’s kind of the opposite of how I write ordinary tests, where I pick challenging but representative examples. Often I’m just checking some basic property, like “function f is the inverse of function g” for round tripping data through a different representation. Sometimes it’s just “function f does not divide by zero.”. Hypothesis seems to have a library of corner cases that it checks first. If you don’t say you can’t handle floats above 2^53, it will try 2^53, for instance. (That’s the threshold where doubles stop being able to represent every integer.) Often the result of running hypothesis is not a code change but a more precise description of what inputs are valid.

Re: Hypothesis: Property-Based Testing for Python

#77

Post by sevensor »

tybug wrote: post #31
eru wrote: post #29

Earlier quoted context omitted.

> I don't know anything about Hypothesis in Python, but I don't think this is true in general. The reason is because the generator can actually inspect your runtime binary and see what branches are being triggered and try to find inputs that will cause all branches to be executed. The author of Hypothesis experimented with this feature once, but people usually want their unit tests to run really quickly, regardless o…

(Hypothesis maintainer here) Yup, a standard test suite just doesn't run for long enough for coverage guidance to be worthwhile by default. That said, coverage-guided fuzzing can be a really valuable and effective form of testing (see eg https://hypofuzz.com/ ).

Thank you, Hypothesis is brilliant!

Re: Hypothesis: Property-Based Testing for Python

#78

Post by movpasd »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

You can either use the @example decorator to force Hypothesis to check an edge case you've thought of, or just let Hypothesis uncover the edge cases itself. Hypothesis won't fail a test once and then pass it next time, it keeps track of which examples failed and will re-run them. The generated inputs aren't uniformly randomly distributed and will tend to check pathological cases (complex symbols, NaNs, etc) with priority.

You shouldn't think of Hypothesis as a random input generator but as an abstraction over thinking about the input cases. It's not perfect: you'll often need to .map() to get the distribution to reflect the usage of the interface being tested and that requires some knowledge of the shrinking behaviour. However, I was really surprised how easy it was to use.

Re: Hypothesis: Property-Based Testing for Python

#79

Post by teiferer »

eru wrote: post #24
teiferer wrote: post #16

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the e…

> 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert No. That's one valid approach, especially if you have a simpler alternative implementation. But testing against an oracle is far from the only property you can check. For your example: suppose you have implemented an add function for your fancy new data type (perhaps it's a crazy vector/tensor th…

> a + b == b + a

> a + (b + c) = (a + b) + c

> a + (-a) == 0

Great! Now I have a stupid bug that always returns 0, so these all pass, and since I didn't think about this case (otherwise I'd not have written that stupid bug in the first place), I didn't add a property about a + b only being 0 if a == -b and boom, test is happy, and there is nothing that the framework can do about it.

Coming up with those properties is hard for real life code and my main gripe with formal methods based approaches too, like model checking or deductice proofs. They move the bugs from the (complicated) code to the list of properties, which ends up just as complicated and error prone, and is entirely un...tested.

Contrast that with an explicit dead-simple test. Test code doesn't have tests. It needs to be orders of magnitudes simpler than the system it's testing. Its correctness must be obvious. Yes, it is really hard to write a good test. So hard that it should steer how you architect your system under test and how you write code. How can I test this to have confidence that it's correct? That must be a guiding principle from the first line of code. Just doing this as an afterthought by playing lottery and trying to come up with smart properties after the fact is not going to get you the best outcome.

Re: Hypothesis: Property-Based Testing for Python

#80

Post by arnsholt »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

I've only used PBT a few times, but when it fits it's been extremely useful. A concrete example from my practice of what has been pointed out in this thread, that you want to properties of your function's output rather than the output itself: I was implementing a fairly involved algorithm (the Zhang-Shasha edit distance algorithm for ordered trees), and PBT was extremely useful in weeding out bugs. What I did was writing a function that generated random tree structures in the form I needed for my code, and tested the four properties that all distance functions should have:

1. d(x, x) = 0 for all x 2. d(x, y) >= 0 for all x, y 3. d(x, y) = d(y, x) for all x, y 4. d(x, z) Especially the triangle inequality check weeded out some tricky corner cases I probably wouldn't have figured out on my own. Some will object that you're not guaranteed to find bugs with this kind of random-generation strategy, but if you blast through a few thousand cases every time you run your test suite, and the odd overnight run testing a few million, you quickly get fairly confident that the properties you test actually hold. Of course any counterexamples the PBT finds should get lifted to regression tests in addition, to make sure they're always caught if they crop up again. And as with any testing approach, there are no guarantees, but it adds a nice layer of defense in depth IMO.

Post reply on HN