Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

31–40 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#31

Post by tybug »

eru wrote: post #29
ChadNauseam wrote: post #27

Earlier quoted context omitted.

> Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does. Not true. For example, if `f` is `+`, you can assert that f(x,y) == f(y,x). Or that f(x, 0) == x. Or that f(x, f(y, z)) == f(f(x, y)…

> I don't know anything about Hypothesis in Python, but I don't think this is true in general. The reason is because the generator can actually inspect your runtime binary and see what branches are being triggered and try to find inputs that will cause all branches to be executed. The author of Hypothesis experimented with this feature once, but people usually want their unit tests to run really quickly, regardless o…

(Hypothesis maintainer here)

Yup, a standard test suite just doesn't run for long enough for coverage guidance to be worthwhile by default.

That said, coverage-guided fuzzing can be a really valuable and effective form of testing (see eg https://hypofuzz.com/).

Re: Hypothesis: Property-Based Testing for Python

#32

Post by eru »

sunshowers wrote: post #14
klntsky wrote: post #11

It seems to only implement a half of QuickCheck idea, because there is no counterexample shrinking. Good effort though! I wonder how hard would it be to derive generators for any custom types in python - probably not too hard, because types are just values

Shrinking is by far the most important and impressive part of Hypothesis. Compared to how good it is in Hypothesis, it might as well not exist in QuickCheck. Proptest in Rust is mostly there but has many more issues with monadic bind than Hypothesis does (I wrote about this in https://sunshowers.io/posts/monads-through-pbt/ ).

Python's Hypothesis has some very clever features to deal with shrinking past a monadic bind.

If I remember right, it basically uses a binary 'tape' of random decisions. Shrinking is expressed as manipulations of that tape. Your generators (implicitly) define a projection from that tape to your desired types. Shrinking an early part of the tape, leave the later sub-generators to try and re-use the later parts of the tape.

That's not guaranteed to work. But it doesn't have to work reliably for every shrink operation the library tries! It's sufficient, if you merely have a good-enough-chance to recover enough of the previous structure to trigger the bug again.

Re: Hypothesis: Property-Based Testing for Python

#33

Post by sunshowers »

eru wrote: post #28
sunshowers wrote: post #6

Earlier quoted context omitted.

The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. A more complex kind of PBT is if you have two implementations of an algorithm or data structure, one that's fast but tricky and the other one slow but easy to verify. (Say, quick sort vs bubble sort.) Generate data or operations ra…

> The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest. The absolute simplest I can think of is just running your functionality on so…

Speaking for myself — those are definitely all simpler cases, but for me I never found them compelling enough (beyond the "it doesn't crash" property). For me, the simplest case that truly motivated PBT for me was roundtrip serialization. Now I use PBT quite a lot, and most of them are either serialization roundtrip or oracle/model-based tests.

Re: Hypothesis: Property-Based Testing for Python

#34

Post by locknitpicker »

teiferer wrote: post #16

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the e…

> (...) but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does.

I think you're manifesting some misconceptions and ignorance about property-based testing.

Property-based testing is still automated testing. You still have a sut and you still exercise it to verify and validate invariants. This does not change.

The core trait of property-based testing is that instead of having to define and maintain hard coded test data to drive your tests, which are specific realizations of the input state, property-based testing instead focuses on generating sequences of randomly-generated input data, and in the event of a test failing it follows up with employing reduction strategies to distil input values that pinpoint minimum reproducible examples.

As a consequence, tests don't focus on which specific value a sut returns when given a specific input value. Instead, they focus on verifying more general properties of a sut.

Perhaps the main advantage of property-based testing is that developers don't need to maintain test data anymore, and this tests are no longer be green just because you forgot to update the test data to cover a scenario or to reflect an edge case. Developers instead define test data generators, and the property-based testing framework implements the hard parts such as the input distillation step.

Property-based testing is no silver bullet though.

> Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of.

Your comment completely misses the point of property-based testing. You still need to exercise your sut to cover scenarios. Where property-based testing excels is that you no longer have to maintain curated sets of test data, or update them whenever you update a component. Your inputs are already randomly generated following the strategy you specified.

Re: Hypothesis: Property-Based Testing for Python

#35

Post by sunshowers »

eru wrote: post #32
sunshowers wrote: post #14

Earlier quoted context omitted.

Shrinking is by far the most important and impressive part of Hypothesis. Compared to how good it is in Hypothesis, it might as well not exist in QuickCheck. Proptest in Rust is mostly there but has many more issues with monadic bind than Hypothesis does (I wrote about this in https://sunshowers.io/posts/monads-through-pbt/ ).

Python's Hypothesis has some very clever features to deal with shrinking past a monadic bind. If I remember right, it basically uses a binary 'tape' of random decisions. Shrinking is expressed as manipulations of that tape. Your generators (implicitly) define a projection from that tape to your desired types. Shrinking an early part of the tape, leave the later sub-generators to try and re-use the later parts of the…

I've always wondered if there could be a small machine learning model trained on shrinking.

Re: Hypothesis: Property-Based Testing for Python

#37

Post by wodenokoto »

eru wrote: post #18
NortySpock wrote: post #2

I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "sever…

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

Re: Hypothesis: Property-Based Testing for Python

#38

Post by 12_throw_away »

teiferer wrote: post #16

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the e…

> Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b

Not really, no, it's right there in the name: you should be testing properties (you can call them "invariants" if you want to sound fancy).

In the example of testing an addition operator, you could test:

1. f(x,y) >= max(x,y) if x and y are non-negative

2. f(x,y) is even iff x and y have the same parity

3. f(x, y) = 0 iff x=-y

etc. etc.

The great thing is that these tests are very easy and fast to write, precisely because you don't have to re-model the entire domain. (Although it's also a great tool if you have 2 implementations, or are trying to match a reference implementation)

Re: Hypothesis: Property-Based Testing for Python

#39

Post by vlovich123 »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

Either your code shouldn’t fail or the apostrophe isn’t a valid case.

In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions.

In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).

Re: Hypothesis: Property-Based Testing for Python

#40

Post by jcattle »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

As far as I remember, hypothesis tests smartly. Which means that possibly problematic strings are tested first. It then narrows down which exact part of the tested strings caused the failure.

So it might as well just throw the kitchen sink at the function, if it handles that: Great, if not: That string will get narrowed down until you arrive at a minimal set of failing inputs.

Post reply on HN