Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

buttondown.com

Post reply on HN
Print view View original

31–40 of 90 posts

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#31

Post by pfdietz »

esafak wrote: post #5

I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.

Tests should be optimized to find bugs. Tests that you have already run have a lower chance of doing that (they only find regressions); tests with novel inputs are preferred. And since writing tests manually is so expensive, this means automatic test input generation. How do you determine if such tests pass? Properties.

In practice, property based testing fails because the organization is not actually interested in delivering correct code. "This bug will never happen in practice so we won't fix it." "If we fix this, we may change some incorrect behavior some customer is depending on." And once that happens, PBT is useless, because it will keep finding that "don't fix" bug over and over.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#32

Post by bluGill »

Akronymus wrote: post #20
alkonaut wrote: post #13

Earlier quoted context omitted.

I think it goes without saying. PBT shouldn't be random-random (e.g. use a timestamp or cryptographic seed), it should be deterministically pseudorandom if it uses random values. You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't).

I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failures in code that they don't understand.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#33

Post by testthetest »

tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

Yes! We use a version of this in our end-to-end Playwright scripts as well, because we want our tests to be both:

1) lightweight, because most of our test suites run on production infrastructure and can’t afford to run them constantly

2) "creative", to find bugs we hadn’t considered before

Probabilistic test scenarios allow us to increase the surface we're testing without needing to exhaustively test every scenario.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#34

Post by Akronymus »

bluGill wrote: post #32
Akronymus wrote: post #20

Earlier quoted context omitted.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failur…

That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basically "hard coding" one of the failure cases)

But yeah, probabilistic testing isn't perfect.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#35

Post by chriswarbo »

tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

Indeed, I've had a property fail due to `Char.isWhitespace` disagreeing with a `\s` regex over whether Mongolian Vowel Separator counts as whitespace (it's not any more, but was prior to Unicode 6.3, according to https://unicode-explorer.com/c/180E )

It's been said (I think by Hughes?) that the causes of property failures tend to be spread equally between buggy code, buggy property and buggy generator.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#36

Post by bpshaver »

bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

Does the article he links to towards the end of the article address your concerns?

> Without complex input spaces, there's no explosion of edge cases, which minimizes the actual benefit of PBT. The real benefits come when you have complex input spaces. Unfortunately, you need to be good at PBT to write complex input strategies. I wrote a bit about it here...

Here's the link: https://www.hillelwayne.com/post/property-testing-complex-in...

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#37

Post by chipsrafferty »

bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

Indeed! The article says PBT is useful, but can't provide any examples of how :/

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#38

Post by Sukera »

metalrain wrote: post #22

I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. For then it's not clear how would one derive the answer from the generated inputs, that is what code is for. But PBT can be great for pruning out crashes you don't expect while parsing.

For aggregation-like things, the interesting properties are often about the properties of the accumulation function, and not the entire aggregation, which should then be correct by extension. So for your `sum` example, you'd use PBT to test that your `+` works first, and only then coming up with things that should hold on top of that when repeatedly applying your operation. For example, once you have a list of numbers and know it's sum, adding one additional number to the list and taking the sum again should lead to the same result as if you had added the number to the sum directly (barring non-associative shenanigans like with floating point - but that should have already been found in the addition step ;) ).

There's a bunch of these kinds of patterns (the above was inspired by [0]) that are useful in practice, but unfortunately rarely talked about. I suppose that's because most people end up replicating their TDD workflows and just throwing more randomness at it, instead of throwing more properties at their code.

[0] https://fsharpforfunandprofit.com/posts/property-based-testi...

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#39

Post by andreareina »

djoldman wrote: post #4

Isn't unit testing a subset of property testing? Seems like a unit test tests specific input and property testing tests more than one input.

They’re different, complementary approaches. Example based testing has a known input and output. Unless you have an oracle (not usually the case) you won’t know the exact output you should get from a generated input. So instead you depend on something that should be true no matter the input. e.g. if you have a serialize/deserialize pair then `deserialize(serialize(x))` better equal `x`.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#40

Post by IanCal »

bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

I used it for running a series of "execute this api call".

If it's valid for a user to do, you can make a list and have it do those in sequence.

I had this for a UI library. It could call the functions to add and create the library and then afterwards would move through it. It was for the BBC so on TVs and could move u/d/l/r - the logic was regardless of the UI if you moved right and the focus changed then moving left should bring you back to where you were (u/d the same, etc).

That's tricky, yet being able to write

FOR ANY ui a person can construct

FOR ANY path a user takes through it

WHEN a user presses R

AND the focus changes

THEN when the user presses L

THEN the user is on the item they were before

Was actually quite easy to write and yet insanely powerful.

The one that really convinced me on PBT was one in this library where it found the bug, and the bug had an explicit test for it and it was explicitly in the spec but the spec was inconsistent! The spec was broken, and nobody had noticed.

Another that drove out a lot of bugs was similar but was that regardless of how many ui changes we made and how many movements the user made, we always had something in focus.

Anyway, the big thing here I want to stress is a series of API calls and asserting something at the end or all the way through.

Side note - oh my this is so long ago, 15 years ago building a new PBT tool in actionscript

Post reply on HN