Great project. I used it a lot, but now I mostly prefer ad hoc generators. Hypothesis combinators quickly become unmaintainable mess for non-trivial objects. Also, shrinking is not such a big deal when you can generate your data in a complexity-sorted order.
Hypothesis: Property-Based Testing for Python
81–90 of 164 posts
Re: Hypothesis: Property-Based Testing for Python
#82Great project. I used it a lot, but now I mostly prefer ad hoc generators. Hypothesis combinators quickly become unmaintainable mess for non-trivial objects. Also, shrinking is not such a big deal when you can generate your data in a complexity-sorted order.
What is complexity sorted order?
Re: Hypothesis: Property-Based Testing for Python
#83Earlier quoted context omitted.
The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. A more complex kind of PBT is if you have two implementations of an algorithm or data structure, one that's fast but tricky and the other one slow but easy to verify. (Say, quick sort vs bubble sort.) Generate data or operations ra…
> The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest. The absolute simplest I can think of is just running your functionality on so…
> The absolute simplest I can think of is just running your functionality on some randomly generated input and seeing that it doesn't crash unexpectedly.
For this use case, we've found it best to just use a fuzzer, and work off the tracebacks.
That being said, we have used hypothesis to test data validation and normalizing code to decent success. We use on a one-off basis, when starting something new or making a big change. We don't run these tests everyday.
Also, I don't like how hypothesis integrates much better with pytest than unittest.
Re: Hypothesis: Property-Based Testing for Python
#84[dead]
Re: Hypothesis: Property-Based Testing for Python
#85Earlier quoted context omitted.
I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?" I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.
Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default. But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem. Also, even with Hypothesis you can and…
> But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code
Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly?
Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.
Re: Hypothesis: Property-Based Testing for Python
#86Earlier quoted context omitted.
Shrinking is by far the most important and impressive part of Hypothesis. Compared to how good it is in Hypothesis, it might as well not exist in QuickCheck. Proptest in Rust is mostly there but has many more issues with monadic bind than Hypothesis does (I wrote about this in https://sunshowers.io/posts/monads-through-pbt/ ).
Python's Hypothesis has some very clever features to deal with shrinking past a monadic bind. If I remember right, it basically uses a binary 'tape' of random decisions. Shrinking is expressed as manipulations of that tape. Your generators (implicitly) define a projection from that tape to your desired types. Shrinking an early part of the tape, leave the later sub-generators to try and re-use the later parts of the…
> Shrinking is expressed as manipulations of that tape.
How do you do that in general? I can't find any documentation on that.
Re: Hypothesis: Property-Based Testing for Python
#87It seems to only implement a half of QuickCheck idea, because there is no counterexample shrinking. Good effort though! I wonder how hard would it be to derive generators for any custom types in python - probably not too hard, because types are just values
The way it does counterexample shrinking is the most clever part of Hypothesis.
Do you have a reference where it is explained? It's not part of the docs as far as I can tell
Re: Hypothesis: Property-Based Testing for Python
#88Property based testing is fantastic. Why is it not more popular? My theory is that only code written in functional languages has complex properties you can actually test. In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.
I think testing culture in general is suffering because the most popular styles/runtimes don’t support it easily.
Most apps (at least in my part of the world) these days are absolutely peppered with side effects. At work our code is mostly just endpoints that trigger tons of side effects, then return some glob of data returned from some of those effects. The joys of micro services!!
If you’re designing from the ground up with testing in mind, you can make things super testable. Defer the actual execution of side effects. Group them together and move local biz logic to a pure function. But when you have a service that’s just a 10,000 line tangle of reading and writing to queues, databases and other services, it’s really hard to ANY kind of testing.
I think that’s why unit testing and full on browser based E2E testing are popular these days. Unit testing pretends the complexity isn’t there via mocks, and lets you get high test coverage to pass your 85% coverage requirement. Then the E2E tests actually test user stories.
I’m really hoping there’s a shift. There are SO many interesting and comprehensive testing strategies available that can give you such high confidence in your program. But it mostly feels like an afterthought. My job has 90% coverage requirements, but not a single person writes useful tests. We have like 10,000 unit tests literally just mocking functions and then spying on the mocked return.
For anybody wanting to see a super duper interesting use of property based testing, check out “Breaking the Bank with test contract”, a talk by Allen Rohner. He pretty much uses property based testing to verify that mocks of services behave identically to the actual services (for the purpose of the program) so that you can develop and test against those mocks. I’ve started implementing a shitty version of this at work, and it’s wicked cool!!
Re: Hypothesis: Property-Based Testing for Python
#89Is there something this nice for JS, with the decorators like that?
No decorators, but fast-check has add-ons to various test frameworks. E.g. if you use Vitest you can write: import { test, fc } from '@fast-check/vitest' test.prop([fc.array(fc.double())])('sort is correct', (lst) => { expect(lst).toEqual(lst.toSorted()) }) https://www.npmjs.com/package/@fast-check/vitest?activeTab=r...
Fast check is fantastic!! I found it to be pretty verbose but I think that’s just a typescript limitation. It’s VERY well typed, which was a nice surprise. Such a great library. Shrinking, model based testing, it’s really comprehensive
Re: Hypothesis: Property-Based Testing for Python
#90Earlier quoted context omitted.
In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…
But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?
If you know it will fail on apostrophe you should have a specific test for that. However if that detail is burried in some function 3 levels deep that you don't even realize is used you wouldn't write the test or handle it even though it matters. This should find those issuses too.