Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

111–120 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#111

Post by dragonwriter »

wodenokoto wrote: post #37
eru wrote: post #18

Earlier quoted context omitted.

In addition to what other people have said: > [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...] You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful. As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about t…

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

> But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

If you just naively treat it as a string and let hypothesis generate values, sure. Which is better than if you are doing traditional explicit unit testing and haven’t explicitly defined apostrophes as a concern.

If you do have it (or special characters more generally) as a concern, that changes how you specify your test.

Re: Hypothesis: Property-Based Testing for Python

#112

Post by dragonwriter »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

> Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all.

You don't need that, that’s just a convenient shortcut when you do have a function that does that (but, I would argue, a not-great example for precisely that reason.)

All you actually need is the ability to verify the property (or set of properties) that you are testing in the output. You could replace the test in that sample with something like this, which I think is a better illustration of what is going on (this is somewhat simplified, rather than the length test you really want to test that it has the same set:

  @given(st.lists(st.integers() | st.floats()))
  def test_sort_correct(lst):
    result = my_sort(lst)
    
    # result has the same unique items as original list
    assert set(result) | set(lst) 

    # for each item, result has the same number of copies as original
    assert all(
      result.count(item) == lst.count(item)
      for item in set(result)
    ) 

    # result is sorted
    assert all(result[n]

Re: Hypothesis: Property-Based Testing for Python

#113

Post by dkarl »

rmunn wrote: post #7

I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (…

> Once my property tests were running overnight and not finding any failures

Quickly addressing the time it takes to run property-based tests, especially in Python: it's extremely helpful to make sure that running subsets of unit tests is convenient and documented and all your developers are doing it. Otherwise they may revolt against property-based testing because of the time it takes to run tests. Again, especially in Python.

Re: Hypothesis: Property-Based Testing for Python

#114

Post by akshayshah »

jamesponddotco wrote: post #105

Never heard of “property-based testing” before. Coming from Go, I mostly use table and fuzzy tests. Is this approach something that makes sense in Go as well? I see at least one package[1][2] for Go, but I wonder if the approach itself makes sense for the language. [1]: https://github.com/flyingmutant/rapid [2]: Oh, I just found out about testing/quick.

Rapid is excellent. It also integrates with the standard library's fuzz testing, which is handy to persist a high-priority corpus of inputs that have caused bugs in the past.

Testing/quick is adequate for small things and doesn't introduce new dependencies, but it's also frozen. Many years ago, the Go team decided that PBT is complex enough that it shouldn't be in stdlib.

Here's a small example of a classic PBT technique used in a very practical Go project: https://github.com/connectrpc/connect-go/blob/cb2e11fb88c9a6...

Re: Hypothesis: Property-Based Testing for Python

#115

Post by DRMacIver »

spooky_deep wrote: post #47

Property based testing is fantastic. Why is it not more popular? My theory is that only code written in functional languages has complex properties you can actually test. In imperative programs, you might have a few utils that are appropriate for property testing - things like to_title_case(str) - but the bulk of program logic can only be tested imperatively with extensive mocking.

How popular do you want it to be?

The Python survey data (https://lp.jetbrains.com/python-developers-survey-2024/) holds pretty consistently at 4% of Python users saying they use it, which isn't as large as I'd like, but given that only 64% of people in the survey say they use testing at all isn't doing too badly, and I think certainly falsifies the claim that Python programs don't have properties you can test.

Re: Hypothesis: Property-Based Testing for Python

#116

Post by NortySpock »

fwip wrote: post #3
NortySpock wrote: post #2

I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand. So rather than try to learn to black boxes at the same time , I fall back to "sever…

I think the easiest way is to start with general properties and general input, and tighten them up as needed. The property might just be "doesn't throw an exception", in some cases. If you find yourself writing several edge cases manually with a common test logic, I think the @example decorator in Hypothesis is a quick way to do that: https://hypothesis.readthedocs.io/en/latest/reference/api.ht...

Thanks, the "does not throw an exception" property got my mental gears turning in terms of how to get started on this, and from there I can see how one could add a few more properties as one goes along.

Appreciate you taking the time to answer.

Re: Hypothesis: Property-Based Testing for Python

#117

Post by dragonwriter »

teiferer wrote: post #16

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the e…

Your comment is downvoted currently, but I think it has value in the discussion (despite being wrong in literally every respect) because it shows the immense misleading power of a single extraordinarily poorly chosen headline example on the project page. Testing a sorting function against the results of the sorted builtin is concise, and technically correct, but (even though there are situations where it would be exactly the right thing to do) it is completely misleading to anyone new to the concept when it comes to indicating what property-based testing is all about.

> It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it.

It does not, and it would be next to worthless if it did. It requires being able to define the properties required of the SUT and right implement code that can refute them if they are not present (the name "hypothesis" for this library is, in fact, a reference to that; PBT treats the properties of code as a hypothesis, and attempts to refute it.)

> but in the end of the day, you somehow need to derive the expected outputs from input arguments, just like your SUT does.

No, see my reimplementation of the sorting example without resorting to the builtin (or any) sorting function other than the one being tested:

https://news.ycombinator.com/item?id=45825482

> Despite some anecdata in the comments here, the chances are slim that this approach will find edge cases that you couldn't think of.

You may think this based on zero experience, but I have seen no one who has tried hypothesis even once who has had that experience. Its actually very good at finding edge cases.

> You as the developer know the implementation and have a chance of coming up with the edge cases.

You as the developer have a very good chance of finding the same edge cases when writing tests for your own code that you considered when writing the code. You have much less chance of finding edge case when writing tests that you missed when writing code. You can incorporate the knowledge of probable edge cases you have when crafting Hypothesis tests just as with more traditional unit tests—but with traditional unit tests you have zero chance of finding the edge cases you didn't think of. Hypothesis is, actually, quite good at that.

> Random tests are just playing the lottery, replacing thinking hard.

Property-based testing doesn’t replace the thinking that goes into traditional unit testing, it just acts as a force multiplier for it.

Re: Hypothesis: Property-Based Testing for Python

#118

Post by FuckButtons »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

Mostly I found making composite strategies most helpful. I had a lot of fiddly details to get right when validating some dsp algorithms and the intermediate data structures they required, and for that it was very helpful, mostly I used property testing with some kind of composite strategy for generating valid inputs.

Re: Hypothesis: Property-Based Testing for Python

#119

Post by travisjungroth »

wongarsu wrote: post #74
lgas wrote: post #71

Earlier quoted context omitted.

Always returning the empty list meets your spec.

Good point. I suppose we should add "number of input elements equals number of output elements" and "every input element is present in the output". Translated in a straightforward test that still allows my_sort([1,1,2]) to return [1,2,2], but we have to draw the line somewhere

Just use Counter and if the objects aren’t hashable, use the count of IDs. Grab this before calling the function, in case the function is destructive. Check it against the output.

Add in checking each item is less than or equal to its successor and you have the fundamental sort properties. You might have more, like stability.

Re: Hypothesis: Property-Based Testing for Python

#120

Post by dragonwriter »

imiric wrote: post #61
ibizaman wrote: post #53

Earlier quoted context omitted.

I actually used property testing very successfully to test a DB driver and a migration to another DB driver in Go. I wrote up about it here https://blog.tiserbox.com/posts/2024-02-27-stateful-property...

Thanks for sharing! Your article illustrates well the benefits of this approach. One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones. This means that bugs are much more likely, they're more difficult to maintain, etc. You do mention that it's a lot of code, but I wonder if the complexity is worth it in the long run. I suppose that since testing these scenari…

> One drawback I see is that property-based tests inevitably need to be much more complex than example-based ones.

I don’t think that’s true, I just think the complexity is more explicit (in code) rather than implicit (in the process of coming up with examples). Example-based testing usually involves defining conditions and properties to be tested, then involves constructing sets of examples to test them and which attempt to anticipated edge cases from the description of the requirements (black box) or from knowledge of how the code is implemented (white box).

Property based testing involves defining the conditiosn and properties, writing code that generates the conditions and for each property writing a bit of code that can refute it by passing if and only if it is true of the subject under test for a particular set of inputs.

With a library like Hypothesis which both has good generators for basic types and good abstractions for combining and adapting generators, the latter seems to be less complex overall, as well as moving the complexity into a form where it is explicit and easy to maintain/adapt, whereas adapting example-based tests to requirements changes involves either throwing out examples and starting over or revalidating and updating examples individually.

Post reply on HN