Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

buttondown.com

Post reply on HN
Print view View original

71–80 of 90 posts

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#71

Post by Jtsummers »

chriswarbo wrote: post #59
bluGill wrote: post #10

Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…

For code that's more "business logic" rather than "algorithmic", I find the following helpful: - Despite the terrible tutorial examples, PBT isn't about running one function on an arbitrary input, then trying to think of assertions about the result. Instead, focus on ways that different parts of your production code fits together, what assumptions are being made at each point, etc. - You don't need to plug random inp…

https://hypothesis.readthedocs.io/en/latest/stateful.html

That last test style you describe can be done with Hypothesis. I've had some good success testing both Python programs and programs written in other languages that could be driven from Python with it. Like a server using gRPC (or CORBA once) as an interface, driven by tests written in Python imitating client behavior.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#72

Post by zelphirkalt »

bluGill wrote: post #32
Akronymus wrote: post #20

Earlier quoted context omitted.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

That is the one thing about PBT that worries me. I can write code and all tests pass, then next week the edge case I missed randomly is hit by a coworker who now has to figure out why their change broke my code (it didn't). I can tell you from experience that random failures cause loss of trust. People learn to ignore failures and just keep hitting rebuild until the tests pass. People will not investigate test failur…

Usually though the failure case inputs should be stored, so that once the test fails, it will fail again, no matter how often you hit the run or rebuild button.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#73

Post by nottorp »

rented_mule wrote: post #69
nottorp wrote: post #21

Earlier quoted context omitted.

That's Unicode, which has a whole section of hell dedicated to it.

Is the root of the problem Unicode? Or is the root of the problem the complexity of the union of written human languages? To the extent that it's the latter, Unicode is just the messenger.

I would shed a tear but then I remember that they have not one, not two but four canonical forms...

And what do they do about it? They add more emoji.

Besides, even if it's justified, it's still sections 7.1-A to 7.3-D of hell.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#74

Post by alkonaut »

Akronymus wrote: post #20
alkonaut wrote: post #13

Earlier quoted context omitted.

I think it goes without saying. PBT shouldn't be random-random (e.g. use a timestamp or cryptographic seed), it should be deterministically pseudorandom if it uses random values. You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

When would that be preferable to have nondeterministic random?

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#75

Post by Akronymus »

alkonaut wrote: post #74
Akronymus wrote: post #20

Earlier quoted context omitted.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

When would that be preferable to have nondeterministic random?

I don't think I have said anything about it being nondeterministic.

You can have a prng seeded from something like the commit hash and have prngs generate the test cases. That still can fail on 1/10 tests for a particular run.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#76

Post by sn9 »

tikhonj wrote: post #14

I wrote some property-based tests for a parsing library. Eventually one of the tests for case-insensitive parsing failed... because it hit the letter "DŽ" which has upper, lower and title-case. The funny thing is that the parsing library was correct and it was the test property that was wrong—but I still learned about an edge case I had never considered! This has been a common pattern for "simpler" property-based test…

There are techniques to discover properties of your function as written, which can then tell you if you've written the function you intended to write: https://www.fuzzingbook.org/html/DynamicInvariants.html (also at https://www.debuggingbook.org/html/DynamicInvariants.html)

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#77

Post by Jtsummers »

alkonaut wrote: post #74
Akronymus wrote: post #20

Earlier quoted context omitted.

> You shouldn't be able to run it 10 times and get 9 pass and one failure. It's either 10 passes or 10 failures. With property based testing, it actually CAN be 9 passes and 1 failure, because that one single fail can be hitting an edge case the others just aren't. In fact, only a few failures are more likely than it being all failures

When would that be preferable to have nondeterministic random?

With PBT you generate an arbitrary number of inputs, ideally they all pass. However it's entirely possible that you have an error in your program and the property only holds for some inputs, but not all. In that case, since each execution starts with a different seed (unless you provide a specific seed, in which case the generated inputs should be exactly the same each time), you may have some executions that always pass, some that always fail, and others that are in between (have a mix of passes and failures). This is expected.

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#78

Post by moi2388 »

recroad wrote: post #25
moi2388 wrote: post #12

Why are we even testing to begin with, and not using theorem provers like lean to prove without any doubt that our commutative function is indeed commutative?

Can you please explain more and maybe give some examples that resonate with people who don't have the understanding that you do?

An example would be critical systems such as defence and Aerospace, where they use for example Ada Spark to formally proof certain bugs cannot occur

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#79

Post by pfdietz »

chriswarbo wrote: post #67
pfdietz wrote: post #31

Earlier quoted context omitted.

Tests should be optimized to find bugs. Tests that you have already run have a lower chance of doing that (they only find regressions); tests with novel inputs are preferred. And since writing tests manually is so expensive, this means automatic test input generation. How do you determine if such tests pass? Properties. In practice, property based testing fails because the organization is not actually interested in d…

Your last paragraph is using terms like "correct", "fix" and "bug" as if they're absolute, when they're actually relative to some sort of spec (whether formal or informal, written or vibes-based, etc.). If the organisation controls the spec, then it can be perfectly reasonable for them to update that spec to e.g. allow certain behaviours that previously would have been considered bugs. In that case, we update the pro…

The spec becomes "don't break important customers code". How could one possibly formalize that?

Re: Why Property Testing Finds Bugs Unit Testing Does Not (2021)

#80

Post by sundarurfriend »

nottorp wrote: post #23
esafak wrote: post #5

I independently discovered PBT when I was a junior, and suggested we use it. My coworkers rejected it because tests should be predictable, and it's the programmer's job to pick the edge cases.

It's unrelated to this article, but I suppose this motivation is why no one has a Gremlins-like feature any more. It seems to be almost totally forgotten, since the only link I could find is an excerpt from a PalmOS programming book: https://www.oreilly.com/library/view/palm-programming-the/15...

The gremlins turned into monkeys (as they quickly did in the page you shared as well):

https://developer.android.com/studio/test/other-testing-tool...

Post reply on HN