Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

buttondown.com

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#61

Post by chriswarbo »

aswerty wrote: post #47
Akronymus wrote: post #34

Earlier quoted context omitted.

That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basica…

Potentially using the git hash as a seed would make sense, so for a given snapshot of code it is always going to be deterministic. When the git hash changes (i.e. your code) then that would result in a different set of test inputs running. Allowing reproducibility for a given change set.

Using a git hash still has the problem of a co-worker's changes (which alter the git commit) causing an unrelated property to fail.

Hypothesis has a nice option, to pick the seed for each property by hashing that property's code. It's a nice idea, but relies on Python's highly dynamic nature; so may not be easy/possible in other languages (especially compiled ones).

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#62

Post by bluGill »

jononor wrote: post #43
bluGill wrote: post #32

Earlier quoted context omitted.

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failur…

So lock the seed in that case?

If you lock your seed you are worse than unit tests - odds are you are not testing some interesting cases ever. The whole point of PBT is there are some properties that hold in all cases (well technically in the domain of the function inputs), so try random examples to see if I missed something. Generally the domain of all possible function inputs is a very large set such that exhaustive testing is impossible. The more different random seeds we try the better the odds we eventually catch some case that we didn't handle correctly.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#63

Post by bluGill »

bpshaver wrote: post #36
bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

Does the article he links to towards the end of the article address your concerns? > Without complex input spaces, there's no explosion of edge cases, which minimizes the actual benefit of PBT. The real benefits come when you have complex input spaces. Unfortunately, you need to be good at PBT to write complex input strategies. I wrote a bit about it here... Here's the link: https://www.hillelwayne.com/post/property-…

It is a start, but I still feel like I'm not sure how I'd apply that to my own domain.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#64

Post by chriswarbo »

nottorp wrote: post #23
esafak wrote: post #5

I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.

It's unrelated to this article, but I suppose this motivation is why no one has a Gremlins-like feature any more. It seems to be almost totally forgotten, since the only link I could find is an excerpt from a PalmOS programming book: https://www.oreilly.com/library/view/palm-programming-the/15...

That sounds like fuzz testing; which is similar to PBT, but (a) usually checks a single property ("the program doesn't crash") and (b) sends data via the program/system's ordinary input channels (whereas PBT has "white box" access to internals, like unit test do).

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#65

Post by chriswarbo »

djoldman wrote: post #4

Isn't unit testing a subset of property testing? Seems like a unit test tests specific input and property testing tests more than one input.

Yes, if you've got a mixture of unit tests and property tests then you can write them all using a single PBT framework, rather than needing two test frameworks.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#66

Post by pfdietz »

chipsrafferty wrote: post #42
pfdietz wrote: post #7

Earlier quoted context omitted.

I'll give you the good example I've been doing for the last two decades: testing a compiler. The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations? The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compi…

Unfortunately I have no idea what I'm looking at here.

The function constructs two lambda expressions (source code for anonymous functions) that should be equivalent. One has some extra declarations. It then compiles the two lambda expressions and calls the compiled code on the same arguments, and gets different values (which is the bug).

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#67

Post by chriswarbo »

pfdietz wrote: post #31
esafak wrote: post #5

I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.

Tests should be optimized to find bugs. Tests that you have already run have a lower chance of doing that (they only find regressions); tests with novel inputs are preferred. And since writing tests manually is so expensive, this means automatic test input generation. How do you determine if such tests pass? Properties. In practice, property based testing fails because the organization is not actually interested in d…

Your last paragraph is using terms like "correct", "fix" and "bug" as if they're absolute, when they're actually relative to some sort of spec (whether formal or informal, written or vibes-based, etc.). If the organisation controls the spec, then it can be perfectly reasonable for them to update that spec to e.g. allow certain behaviours that previously would have been considered bugs.

In that case, we update the properties to reflect the new spec.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#68

Post by atmavatar »

nottorp wrote: post #21
tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

That's Unicode, which has a whole section of hell dedicated to it.

When the robots rise up and exterminate the human race, Unicode will be one of the reasons we deserve it.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#69

Post by rented_mule »

nottorp wrote: post #21
tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

That's Unicode, which has a whole section of hell dedicated to it.

Is the root of the problem Unicode? Or is the root of the problem the complexity of the union of written human languages? To the extent that it's the latter, Unicode is just the messenger.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#70

Post by skybrian »

crabbone wrote: post #19
skybrian wrote: post #6

Earlier quoted context omitted.

You do need to be able to reproduce a test failure, which can be done by printing the inputs or the random seed used in a way that makes it trivial to rerun it.

Nah. That's not what this is about. In any system worth testing, with property-based testing, the system will take so many steps before it encounters an error that even knowing what the steps were isn't going to be very helpful in reproducing the error. I've been there. It takes many hours to try to guess where the system went wrong to produce the undesirable result, and then you still might not be sure if you are lo…

Yes, timeouts are bad, but that’s true even for regular tests with determininistic input. You’ll get flakes running on different machines, depending on how much load is on them.

To test an entire system with reproducable failures, you probably need something more heavyweight like Antithesis. Property tests are more useful for unit tests.

Post reply on HN