Why Property Testing Finds Bugs Unit Testing Does Not (2021)
1–10 of 90 posts
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#2I haven't used PBT much, but I did once get a lot of mileage out of it, which was when I was implementing a fairly gnarly edit-distance algorithm. In that case, I used PBT to check that the required properties of metric functions (d(x,x)=0, d(x,y)>0, d(x,y)=d(y,x), d(x,z) <= d(x,y)+d(y,z)) held for my implementation, which helped shake out a fair few stupid implementation mistakes.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#3Aw man, I was nodding along with the "most examples suck" section and then... it ended :(
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#4Isn't unit testing a subset of property testing?
Seems like a unit test tests specific input and property testing tests more than one input.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#5I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#6I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.
You do need to be able to reproduce a test failure, which can be done by printing the inputs or the random seed used in a way that makes it trivial to rerun it.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#7Aw man, I was nodding along with the "most examples suck" section and then... it ended :(
I'll give you the good example I've been doing for the last two decades: testing a compiler.
The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations?
The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compiled with different optimization levels should do the same thing." This assumes the randomly generated code doesn't touch undefined behavior in the language spec.
Here's a recent example of a bug found by this approach. The Common Lisp code stimulating the bug has been automatically minimized: https://bugs.launchpad.net/sbcl/+bug/2109837 with the bug fix https://sourceforge.net/p/sbcl/sbcl/ci/1abebf7addda1a43d6d24...
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#8Isn't unit testing a subset of property testing? Seems like a unit test tests specific input and property testing tests more than one input.
Yes, but converting a test to be data-driven adds some complexity (how do you debug just the failures?) and generating the inputs randomly makes it harder to know what properties to assert.
Also, if you never look at the test data, it might give you false confidence about how thoroughly your code is being tested.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#9One can use property testing to automatically generate unit test inputs. The property becomes "does this do something new that I wanted to test for but wasn't?" (or, rather, the negation; inputs "pass" and can be ignored if they do nothing new.) This could be code coverage, or it could be killing code mutants.
Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)
#10Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I write a property for non-trivial functions it not clear at all, and the article stops.