Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

101–110 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#101

Post by agumonkey »

groundhogstate wrote: post #62
ed_blackburn wrote: post #59

I'm using sqlglot to parse hundreds of old mysql back up files to find diffs of schemas. The joys of legacy code. I've found hypothesis to be super helpful for tightening up my parser. I've identified properties (invariants) and built some strategies. I can now generate many more permutations of DDL than I'd thought of before. And I have absolutely confidence in what I'm running. I started off TDD covered the basics.…

You might find schemathesis useful for API testing (which IIRC is built on Hypothesis). Certainly helped me find a stack of unhandled edge cases. YMMV, but once I first set up Schemathesis on one of my MVPs, it took several hours to iron out the kinks. But thereafter, if built into CI, I've found it guards against regression quite effectively.

pretty nice tool, helped us poke into our app in ways we didn't foresee

also pretty happy to see the reports of thousands or api calls that matched the expected responses, that increases the confidence you have in your system

Re: Hypothesis: Property-Based Testing for Python

#102

Post by nyrikki »

thunky wrote: post #85
eru wrote: post #50

Earlier quoted context omitted.

Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default. But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem. Also, even with Hypothesis you can and…

> But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly? Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in p…

Saving the seed in the build artifacts/logs has saved a lot of time for me even with tools like faker.

Re: Hypothesis: Property-Based Testing for Python

#103

Post by theptip »

IanCal wrote: post #44
wodenokoto wrote: post #37

Earlier quoted context omitted.

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

Hypothesis keeps a database of failures to use locally and you can add a decorator to mark a specific case that failed. So you run it, see the failure, add it as a specific case and then that’s committed to the codebase. The randomness can bite a little if that test failure happens on an unrelated branch, but it’s not much different to someone just discovering a bug. edit - here's the relevant part of the hypothesis…

You can also cache the DB across CI runs, which will reduce the randomness (ie failures won’t just disappear between runs).

Re: Hypothesis: Property-Based Testing for Python

#104

Post by iLemming »

bunderbunder wrote: post #99
rmunn wrote: post #7

I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (…

I love them for this, too. Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent. (I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a librar…

> "no non-deterministic tests"

These tests can be made determenistic. Shrinking to the minimal reproducing case, is always deterministic. Also you can [re]start the tests with the specific seed - you typically log the seed, so you can replay the exact failure deterministically - prop-based tests are stochastic by design but reproducible by capture.

Re: Hypothesis: Property-Based Testing for Python

#105

Post by jamesponddotco »

Never heard of “property-based testing” before. Coming from Go, I mostly use table and fuzzy tests.

Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language.

[1]: https://github.com/flyingmutant/rapid

[2]: Oh, I just found out about testing/quick.

Re: Hypothesis: Property-Based Testing for Python

#106

Post by iLemming »

spooky_deep wrote: post #47

Property based testing is fantastic. Why is it not more popular? My theory is that only code written in functional languages has complex properties you can actually test. In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.

> Why is it not more popular?

Probably because it's difficult to build simplistic intuition around them in languages that don't have robust built-in mechanisms that enable that kind of thinking.

For example, gen-testing in Clojure feels simple and intuitive, but if I have to do something similar, hmmm... I dunno, say in Java, I wouldn't even know where to start - the code inevitably would be too verbose; mutable state would obscure invariants; and the required boilerplate alone would cloud the core idea. While in Clojure, immutability-by-default feels transformative for gen-testing; data generation is first-class - generators are just values or functions; composition is natural; debugging is easier; invariants are verifiable; FP mindset already baked in - easier to reason about stateful sequences/event streams; there's far less ceremony overall;

But if I'm forced to write Java and see a good case for prop-based testing, I'd still probably try doing it. But only because I already acquired some intuition for it. That's why it is immensely rewarding to spend a year or more learning an FP language, even if you don't see it used at work or every day.

So, the honest answer to your question would probably be: "because FP languages are not more popular"

Re: Hypothesis: Property-Based Testing for Python

#107

Post by masklinn »

wongarsu wrote: post #74
lgas wrote: post #71

Earlier quoted context omitted.

Always returning the empty list meets your spec.

Good point. I suppose we should add "number of input elements equals number of output elements" and "every input element is present in the output". Translated in a straightforward test that still allows my_sort([1,1,2]) to return [1,2,2], but we have to draw the line somewhere

> we have to draw the line somewhere

Do we? You can pop-count the two lists and checks that those are equal.

Re: Hypothesis: Property-Based Testing for Python

#108

Post by dd82 »

iLemming wrote: post #104
bunderbunder wrote: post #99

Earlier quoted context omitted.

I love them for this, too. Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent. (I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a librar…

> "no non-deterministic tests" These tests can be made determenistic. Shrinking to the minimal reproducing case, is always deterministic. Also you can [re]start the tests with the specific seed - you typically log the seed, so you can replay the exact failure deterministically - prop-based tests are stochastic by design but reproducible by capture.

correct, I've done this at past places to verify rules engine output for variety of inputs.

Re: Hypothesis: Property-Based Testing for Python

#109

Post by dd82 »

NortySpock wrote: post #2

I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "sever…

Essentially this is a good example of parametrized tests, just supercharged with generated inputs.

So if you already have parametrized tests, you're already halfway there.

Re: Hypothesis: Property-Based Testing for Python

#110

Post by sunshowers »

ctenb wrote: post #86
eru wrote: post #32

Earlier quoted context omitted.

Python's Hypothesis has some very clever features to deal with shrinking past a monadic bind. If I remember right, it basically uses a binary 'tape' of random decisions. Shrinking is expressed as manipulations of that tape. Your generators (implicitly) define a projection from that tape to your desired types. Shrinking an early part of the tape, leave the later sub-generators to try and re-use the later parts of the…

> Shrinking is expressed as manipulations of that tape. How do you do that in general? I can't find any documentation on that.

This paper by the Hypothesis authors has some information: https://www.doc.ic.ac.uk/~afd/papers/2020/ECOOP_Hypothesis.p...
Post reply on HN