Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

buttondown.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#41

Post by jononor »

chriswarbo wrote: post #30
metalrain wrote: post #22

I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. For then it's not clear how would one derive the answer from the generated inputs, that is what code is for. But PBT can be great for pruning out crashes you don't expect while parsing.

> I would love to used PBT more, but many tests I write have only one answer per input. Think sum like aggregations. Not quite sure what you mean by "only one answer per input" (that it's a function, i.e. a 1:1 mapping?), but there are of lots of properties that aggregations might typically need to satisfy, e.g. off the top of my head: # Identity element forAll(pre, post) { assertEqual( agg(pre ++ [agg([])] ++ post),…

Numerical aggregates often have the property that their output is in the range min and max of the input.

An aggregate on discrete values my have the property that the output is one of the elements in the input.

It may also have a no-NaN property, or maybe no-NaN unless NaN in input.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#42

Post by chipsrafferty »

pfdietz wrote: post #7
frogulis wrote: post #3

Aw man, I was nodding along with the "most examples suck" section and then... it ended :(

I'll give you the good example I've been doing for the last two decades: testing a compiler. The complexity here is the complete opposite of the simple toy examples. What are the edge cases of an optimizing compiler? How do you even approach them, if they're buried deep in a chain of transformations? The properties are simple things like "the compiler shouldn't crash, the compiled code shouldn't crash, and code compi…

Unfortunately I have no idea what I'm looking at here.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#43

Post by jononor »

bluGill wrote: post #32
Akronymus wrote: post #20

Earlier quoted context omitted.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failur…

So lock the seed in that case?

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#44

Post by jmull »

moi2388 wrote: post #12

Why are we even testing to begin with, and not using theorem provers like lean to prove without any doubt that our commutative function is indeed commutative?

For purely mathematical properties, a purely mathematical technique is probably best.

But "is commutative" is just an example here (one of the topics of this post is how simplistic examples can mislead people as to the usefulness of a given verification technique).

The general point of software verification is to ensure the software "does what I want". But in a very large proportion of cases, people aren't clear on precisely what they want. They could not use a formal method because they could not write a formal specification. A nice thing about unit tests is that you can work through your expectations iteratively and incrementally, broadening and deepening your understanding of exactly what the software should do, capturing each insight along the way in a reusable way.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#46

Post by jononor »

crabbone wrote: post #19
skybrian wrote: post #6

Earlier quoted context omitted.

You do need to be able to reproduce a test failure, which can be done by printing the inputs or the random seed used in a way that makes it trivial to rerun it.

Nah. That's not what this is about. In any system worth testing, with property-based testing, the system will take so many steps before it encounters an error that even knowing what the steps were isn't going to be very helpful in reproducing the error. I've been there. It takes many hours to try to guess where the system went wrong to produce the undesirable result, and then you still might not be sure if you are lo…

Property based testing is incredibly useful for state-free systems/modules. Especially those that have a wide/complex input space. Simple examples would be a general (de)serialization library for something like JSON.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#47

Post by aswerty »

Akronymus wrote: post #34
bluGill wrote: post #32

Earlier quoted context omitted.

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failur…

That's a fair concern. I can only really suggest upping the amount of test cases that are ran when merging so that you get a much more extensive run for that time, and later dial it back. Along with having the seed included in the failure case, so that you can bisect to check what actually broke that test. Also, implementing a standard test alongside the property based on, for all bugs you encounter over time (basica…

Potentially using the git hash as a seed would make sense, so for a given snapshot of code it is always going to be deterministic. When the git hash changes (i.e. your code) then that would result in a different set of test inputs running.

Allowing reproducibility for a given change set.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#48

Post by senderista »

tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

Yup, and many have discovered this is an unexpected benefit of formal methods in general: writing a formal spec forces you to think precisely about requirements, assumptions, and edge cases, independently of whatever benefit formal verification may provide.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#49

Post by senderista »

bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

You can do "white-box" PBT by just asserting all the nontrivial invariants you can think of in your code, and then counting on the generator to force evaluation of those invariants on a representative sample of inputs.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#50

Post by senderista »

aswerty wrote: post #29
alkonaut wrote: post #13

Earlier quoted context omitted.

I think it goes without saying. PBT shouldn't be random-random (e.g. use a timestamp or cryptographic seed), it should be deterministically pseudorandom if it uses random values. You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures.

Just as an anecdotal experience. It doesn't necessarily go without saying. The most memorable discussion I had around PBT was with a colleague (a skip report) who saw "true" randomness as a net benefit and that reproducibility was not a critical characteristic of the test suite (I guess the reasoning was then it could catch things at a later date?). To be honest, it scared the hell out of me and I pushed back pretty…

This dilemma is of course trivially solvable by persisting a (presumably randomly generated) RNG seed with each test run. You just have to ensure that your RNG is configured once with the seed at the beginning of each test run.
Post reply on HN