Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

41–50 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#41

Post by reverius42 »

vlovich123 wrote: post #39
wodenokoto wrote: post #37

Earlier quoted context omitted.

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of test…

I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?"

I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.

Re: Hypothesis: Property-Based Testing for Python

#42

Post by ngruhn »

thdhhghgbhy wrote: post #36

Is there something this nice for JS, with the decorators like that?

No decorators, but fast-check has add-ons to various test frameworks. E.g. if you use Vitest you can write:

    import { test, fc } from '@fast-check/vitest'
    test.prop([fc.array(fc.double())])('sort is correct', (lst) => {
      expect(lst).toEqual(lst.toSorted())
    })
https://www.npmjs.com/package/@fast-check/vitest?activeTab=r...

Re: Hypothesis: Property-Based Testing for Python

#43

Post by aidos »

thdhhghgbhy wrote: post #36

Is there something this nice for JS, with the decorators like that?

Not decorators (or at least not last time I looked) but we use fast-check.

Was already familiar with and using Hypothesis in Python so went in search of something with similar nice ergonomics. Am happy with fast-check in that regard.

https://fast-check.dev/

Re: Hypothesis: Property-Based Testing for Python

#44

Post by IanCal »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

Hypothesis keeps a database of failures to use locally and you can add a decorator to mark a specific case that failed. So you run it, see the failure, add it as a specific case and then that’s committed to the codebase.

The randomness can bite a little if that test failure happens on an unrelated branch, but it’s not much different to someone just discovering a bug.

edit - here's the relevant part of the hypothesis guide https://hypothesis.readthedocs.io/en/latest/tutorial/replayi...

Re: Hypothesis: Property-Based Testing for Python

#45

Post by Balinares »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

No, Hypothesis iterates on test failures to isolate the simplest input that triggers it, so that it can report it to you explicitly.

Re: Hypothesis: Property-Based Testing for Python

#46

Post by IanCal »

dbcurtis wrote: post #4

It’s been quite some time since I’ve been in the business of writing lots of unit tests, but back in the day, I found hypothesis to be a big force multiplier and it uncovered many subtle/embarrassing bugs for me. Recommend. Also easy and intuitive to use.

Huge second.

I’ve never use pbt and failed to find a new bug. I recommended it in a job interview, they used it and discovered a pretty clear bug on their first test. It’s really powerful.

Re: Hypothesis: Property-Based Testing for Python

#47

Post by spooky_deep »

Property based testing is fantastic.

Why is it not more popular?

My theory is that only code written in functional languages has complex properties you can actually test.

In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.

Re: Hypothesis: Property-Based Testing for Python

#48

Post by eru »

sunshowers wrote: post #35
eru wrote: post #32

Earlier quoted context omitted.

Python's Hypothesis has some very clever features to deal with shrinking past a monadic bind. If I remember right, it basically uses a binary 'tape' of random decisions. Shrinking is expressed as manipulations of that tape. Your generators (implicitly) define a projection from that tape to your desired types. Shrinking an early part of the tape, leave the later sub-generators to try and re-use the later parts of the…

I've always wondered if there could be a small machine learning model trained on shrinking.

I'm not sure whether it would be useful, but it would definitely get you a grant (if done academically) or VC money (if done as a company) these days.

Re: Hypothesis: Property-Based Testing for Python

#49

Post by eru »

sunshowers wrote: post #33
eru wrote: post #28

Earlier quoted context omitted.

> The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same. Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest. The absolute simplest I can think of is just running your functionality on so…

Speaking for myself — those are definitely all simpler cases, but for me I never found them compelling enough (beyond the "it doesn't crash" property). For me, the simplest case that truly motivated PBT for me was roundtrip serialization. Now I use PBT quite a lot, and most of them are either serialization roundtrip or oracle/model-based tests.

Oh, yes, I was just listing simple examples. I wasn't trying to find a use case that's compelling enough to make you want to get started.

I got started out of curiosity, and because writing property based tests is a lot more fun than writing example based tests.

Re: Hypothesis: Property-Based Testing for Python

#50

Post by eru »

reverius42 wrote: post #41
vlovich123 wrote: post #39

Earlier quoted context omitted.

Either your code shouldn’t fail or the apostrophe isn’t a valid case. In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions. In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of test…

I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?" I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.

Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default.

But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem.

Also, even with Hypothesis you can and should still write specific tests or even just specific generators to cover specific classes of corners cases you are worried about in more detail.

Post reply on HN