Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypothesis: Property-Based Testing for Python

hypothesis.readthedocs.io

Post reply on HN
Print view View original

151–160 of 164 posts

Re: Hypothesis: Property-Based Testing for Python

#151

Post by tybug »

dragonwriter wrote: post #117
teiferer wrote: post #16

This approach has two fundamental problems. 1. It requires you to essentially re-implement the business logic of the SUT (subject-under-test) so that you can assert it. Is your function doing a+b? Then instead of asserting that f(1, 2) == 3 you need to do f(a, b) == a+b since the framework provides a and b. You can do a simpler version that's less efficient, but in the end of the day, you somehow need to derive the e…

Your comment is downvoted currently, but I think it has value in the discussion (despite being wrong in literally every respect) because it shows the immense misleading power of a single extraordinarily poorly chosen headline example on the project page. Testing a sorting function against the results of the sorted builtin is concise, and technically correct, but (even though there are situations where it would be exa…

(Hypothesis maintainer here) If you have recommendations for a better example on the front page, I'd love to hear them! (I mean this entirely genuinely and non-sarcastically; I agree sorting can give misleading ideas, but it is also concise and well understood by every reader).

Re: Hypothesis: Property-Based Testing for Python

#152

Post by anordin95 »

lwhsiao wrote: post #125
anordin95 wrote: post #123

I am a huge fan of the "Explanations" page of their docs: "These explanation pages are oriented towards deepening your understanding of Hypothesis, including its design philosophy." I feel much more software documentation could greatly benefit from this approach. Describing the "why" and design tradeoffs helps me grok a system far better than the typical quickstart or tutorials which show snippets but offer little un…

You would like the Diátaxis framework: https://diataxis.fr/ That is the structure they (any many others) are following :).

I know it and do like it! Even for those that follow Diataxis, in my experience the "Explanation" sections are often lacking, especially compared to the "How-To" or "Tutorial" ones.

Python is one example that comes to mind. They do have explanations here: https://docs.python.org/3/howto/index.html. And, to be fair, they are generally excellent in my opinion! But they're far from front and center and there's much less overall content compared to the other Diataxis types I think (granted, I haven't rigorously checked).

Re: Hypothesis: Property-Based Testing for Python

#153

Post by eru »

chriswarbo wrote: post #149
teiferer wrote: post #79

Earlier quoted context omitted.

> a + b == b + a > a + (b + c) = (a + b) + c > a + (-a) == 0 Great! Now I have a stupid bug that always returns 0, so these all pass, and since I didn't think about this case (otherwise I'd not have written that stupid bug in the first place), I didn't add a property about a + b only being 0 if a == -b and boom, test is happy, and there is nothing that the framework can do about it. Coming up with those properties is…

> Just doing this as an afterthought by playing lottery and trying to come up with smart properties after the fact is not going to get you the best outcome. This sounds backwards to me. How could you write any tests, or indeed implement any functions, if you don't know any relationships between the arguments/return-value, or the state before/after, or how it relates to other functions, etc.? For the addition example,…

I agree with you. However the grand-parent comment has a point that it's not easy to extract testable properties from code that's already written.

It's much easier to proceed explicitly like you suggest: have some properties in mind, co-develop property tests and code.

Often when I try to solve a problem, I start with a few simple properties and let them guide my way to the solution. Like your deletion example. Or, I already know that the order of inputs shouldn't matter, or that adding more constraints to an optimiser shouldn't increase the maximum, etc.

And I can write down some of these properties before I have any clue about how to solve the problem in question.

---

However, writing properties even after the fact is a learnable skill. You just need to practice, similarly to any other skill.

Re: Hypothesis: Property-Based Testing for Python

#154

Post by ctenb »

rtu8lu wrote: post #93
ctenb wrote: post #82

Earlier quoted context omitted.

What is complexity sorted order?

When you generate randomized data in an order from the simplest to the most complex.

Isn't that very wasteful or difficult to do in practice? If you consider that shrinkers generally take lower numbers to be 'simpler' than higher numbers, complexity-ordering requires you to generate all the numbers from low to high

Re: Hypothesis: Property-Based Testing for Python

#155

Post by dragonwriter »

tybug wrote: post #151
dragonwriter wrote: post #117

Earlier quoted context omitted.

Your comment is downvoted currently, but I think it has value in the discussion (despite being wrong in literally every respect) because it shows the immense misleading power of a single extraordinarily poorly chosen headline example on the project page. Testing a sorting function against the results of the sorted builtin is concise, and technically correct, but (even though there are situations where it would be exa…

(Hypothesis maintainer here) If you have recommendations for a better example on the front page, I'd love to hear them! (I mean this entirely genuinely and non-sarcastically; I agree sorting can give misleading ideas, but it is also concise and well understood by every reader).

The more I think about it the more I think calling it a bad example may be unfair. It can be extremely misleading for someone unfamiliar with the concept coming at it with a particular viewpoint, but I’m less sure, with more time to think, that an example that is better for that wouldn’t be worse in other ways.

I like sorting as an example, and I like that using the built-in is concise, and reimplementing the behavior of an existing function where using the existing function as an oracle is a reasonable thing to do for a test isn't all that uncommon.

I feel like something with a couple of properties described in comments with assertions testing those properties (but where the functionality and properties are familiar enough that it would make a clear connection) would be a bit better, in theory, but I don't have a great particular example to use, and anything done that way will be, at best, somewhat less concise.

Re: Hypothesis: Property-Based Testing for Python

#156

Post by Doxin »

benrutter wrote: post #56

I love the idea of hypothesis! Haven't found a lot of use cases for it yet, I think the quick start example helps explain why. Essentially, you're testing that "my_sort" returns the same as python's standard "sort". Of course, this means you need a second function that acts the same as the function you wrote. In real life, if you had that you probably wouldn't have written the function "my_sort" at all. Obviously it'…

I've used hypothesis in practice for testing a slugify function. When someone signs up for our product our software produces a subdomain based on the organisation name you input. People put all sorts of weird things in that field, and we want to be pretty broad in what we'll accept.

The hypothesis test for this is near trivial. Pass some text to the slugify function, and check it either a) throws an invalid input error (e.g. input too short etc) or b) return a string that's a valid domain name.

Doing this found me so many odd little edge cases around length truncation and idna encoding. The one bug I would've never found myself is the fact that lowercasing "ß" turns it into "ss" which broke truncation, resulting in an invalid domain only if it includes ß and is exactly 64 characters long.

hypothesis is a pretty niche tool, but boy when it's the right tool it's the right tool.

Re: Hypothesis: Property-Based Testing for Python

#157

Post by bunderbunder »

eru wrote: post #144
bunderbunder wrote: post #99

Earlier quoted context omitted.

I love them for this, too. Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent. (I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a librar…

You could just hard code a PRNG seed, and then the test would trivially be deterministic? I'm not sure what they say their objection is, is actually their real objection?

Bluntly, their real objection is something along the lines of, "This isn't how Uncle Bob told me to do it so it must be wrong."

Re: Hypothesis: Property-Based Testing for Python

#158

Post by dragonwriter »

eru wrote: post #153
chriswarbo wrote: post #149

Earlier quoted context omitted.

> Just doing this as an afterthought by playing lottery and trying to come up with smart properties after the fact is not going to get you the best outcome. This sounds backwards to me. How could you write any tests, or indeed implement any functions, if you don't know any relationships between the arguments/return-value, or the state before/after, or how it relates to other functions, etc.? For the addition example,…

I agree with you. However the grand-parent comment has a point that it's not easy to extract testable properties from code that's already written. It's much easier to proceed explicitly like you suggest: have some properties in mind, co-develop property tests and code. Often when I try to solve a problem, I start with a few simple properties and let them guide my way to the solution. Like your deletion example. Or, I…

> However the grand-parent comment has a point that it's not easy to extract testable properties from code that's already written.

True, property-based testing does not solve the problem of deriving the intended behavior of code where behavior is not documented (either by requirements documents, or code comments, or tests that aren't just examples but clearly indicate the general behavior they are confirming, or...)

OTOH, PBT can be used to rapidly test hypotheses about the behavior (though intent is another question) of legacy code, which you are going to need to develop and validate to turn it into code that is maintainable (or even to replace it with something new, if you need to generally be compatible.) Determining whether deviations from a hypothesized behavior are intentional or bugs is still an exercise for the user, though whether the deviations are highly general or narrow to specific cases can help to illuminate that decision, and a library like Hypothesis will help determine that.

Re: Hypothesis: Property-Based Testing for Python

#159

Post by eru »

jcattle wrote: post #40
wodenokoto wrote: post #37

Earlier quoted context omitted.

But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?

As far as I remember, hypothesis tests smartly. Which means that possibly problematic strings are tested first. It then narrows down which exact part of the tested strings caused the failure. So it might as well just throw the kitchen sink at the function, if it handles that: Great, if not: That string will get narrowed down until you arrive at a minimal set of failing inputs.

> Which means that possibly problematic strings are tested first.

Hypothesis uses a the same probability distribution for all the 200 (or so) random cases it generates for a test. The first case has the same distribution as the 200th.

However, Hypothesis gives a pretty large weight in the probability distribution to inputs that are generally 'problematic'. Of course, that's just a heuristic: eg empty lists and 0 and empty strings or strings with apostrophes in them and NaN or infinity often are problematic, but that's just a guess: hypothesis doesn't know anything specific about your code.

The heuristics work remarkably well in practice, though.

Once Hypothesis has found a failing test case, then it tries to shrink it down.

Re: Hypothesis: Property-Based Testing for Python

#160

Post by rtu8lu »

ctenb wrote: post #154
rtu8lu wrote: post #93

Earlier quoted context omitted.

When you generate randomized data in an order from the simplest to the most complex.

Isn't that very wasteful or difficult to do in practice? If you consider that shrinkers generally take lower numbers to be 'simpler' than higher numbers, complexity-ordering requires you to generate all the numbers from low to high

Not really. There are many ways depending on your needs. For example, you can partition your space first, then generate randomly inside each of the subspaces. Let's say I need 200 numbers from -1000 to 999. The first range will be 0 to +99, the second -1 to -100, then +100 to +199, and so on. So, to generate a random number I just need an index and the bounds.
Post reply on HN