Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effective Property-Based Testing

blog.auxon.io

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Effective Property-Based Testing

#31

Post by Vinnl »

mullr wrote: post #22
lowercase1 wrote: post #20

This is a good overview of property based testing but the mentions of Cucumber threw me off. In my job Cucumber seems to add little more than just commenting and sequencing functions, tasks that are better suited to your programming language of choice, while adding overhead and complexity. What am I missing?

To me, Cucumber is a way to write executable specifications that can also be read by non-developers. This can be tremendously powerful if done judiciously, or tremendously pointless if not. This may be because you do not in fact want or need executable specification, or perhaps because there's nobody interested in reading it. I believe that Cucumber is at its best in situations where it's clear to all parties that a…

But if they're only read, not written, by non-developers, wouldn't it make more sense to generate a report from your spec code, rather than using the report as a cumbersome authoring language?

Re: Effective Property-Based Testing

#32

Post by mathw »

I've been trying to push property-based testing at work but there's not a load of enthusiasm showing. I guess it's in part because we work in C# and the best option appears to be FsCheck, which is very inconsistently documented between its F#-native and C#-accessible APIs.

I think there's a strong argument with FsCheck to write all your proptest code in F# just to take advantage of the vastly better generator syntax, but that's a hard sell for a team who mostly don't know F# and aren't convinced proptests are much better anyway. Writing the generators in C# seemed really incredibly tedious. I did start to get the hang of identifying properties to test though. Once you're past the mechanics of "how does this work" that can become much easier.

A load road to travel here, but I kind of gave myself a remit to improve software quality and I do think we need to be looking at this kind of testing to help.

Where do people who are using it find that it offers the most value? I keep feeling that we could really solidify some of our bespoke parsing and serialisation code using this kind of tech.

Re: Effective Property-Based Testing

#33

Post by indoorskier »

mathw wrote: post #32

I've been trying to push property-based testing at work but there's not a load of enthusiasm showing. I guess it's in part because we work in C# and the best option appears to be FsCheck, which is very inconsistently documented between its F#-native and C#-accessible APIs. I think there's a strong argument with FsCheck to write all your proptest code in F# just to take advantage of the vastly better generator syntax,…

FsCheck maintainer here.

I'd very much welcome contributions on documentation, and esp. on approaches of how to keep the C#/F# documentation consistent and still accessible for both types of users. Even if it's just ideas/comments - how would you like the documentation presented? What are examples of excellent C# documentation? We need to balance that with available resources - we don't have a team of ghostwriters to write docs and examples for every language, as you can imagine. I know it's a cliche by this time, but if every user would take a couple minutes to write a paragraph or example where e.g. the C# docs are lacking, it might be in a much better state. From our side, if something is stopping you from contributing in this way, we'd like to hear about it. Addressing that is important.

Separately, I'm surprised you experienced that generators are significantly more tedious to write in C# vs F# - could you open an issue with a few examples of this? This would inform v3.0 where we will stop trying to use tricks to make the F# API accessible, but instead add a bespoke C#/VB.NET API in the FsCheck.Fluent namespace, and separating F#'y bits in FsCheck.FSharp.

Re: Effective Property-Based Testing

#34

Post by zhd »

Smaug123 wrote: post #14

Brilliant article. If I were to add just one thing to the list: metatest. Write a test that asserts that your generated test cases are "sufficiently comprehensive", for whatever value of "sufficiently" you need. In an impure language, this is as easy as having the generator contain a mutable counter for "number of test cases meeting X condition" for whatever conditions you're interested in. For example, say your prop…

Hypothesis can report statistics on user-defined events, as well as the usual timing stuff: https://hypothesis.readthedocs.io/en/latest/details.html#tes...

I'd just check that when you're writing or changing the tests though; for nontrivial conditions it can take a very long time to get neglibible probability of any metatest failing in a given run, and flaky metatests are just as bad as the usual kind.

If this split is particularly important, we'd usually recommend just writing separate tests for data that satisfy A or B; you can even supply the generators with pytest.mark.parametrize if copy-pasting the test body offends.

Re: Effective Property-Based Testing

#35

Post by zhd »

hinkley wrote: post #26

Hypothesis: Property Based Testing is Monte Carlo simulation for model checking.

Yep! And it's also possible to run fuzzers [1, 2] or SAT-based verifiers against the same test harness :-)

1: https://hypofuzz.com/docs/literature.html 2: https://google.github.io/oss-fuzz/getting-started/new-projec...

Re: Effective Property-Based Testing

#36

Post by pfdietz »

amw-zero wrote: post #28
pfdietz wrote: post #25

Turning off shrinking makes little sense to me. If tests are passing there's nothing to shrink. If you are bombarded with so many failures you can't afford to shrink them then more testing is pointless.

The author’s point is that shrinking can take a very long time, and that’s what you should consider turning it off. In my experience this is also true. They’re saying that sometimes it’s better to just see the error in full and try and figure it out.

That is not my experience (doing this sort of testing on compilers). In my experience, the shrinking is usually very fast. When it takes a non-negligible amount of time, debugging the failure would have been hopeless on the non-shrunk input.

What takes the time in my case was simply getting a failure to occur at all. It might take days on a mature compiler before a failure occurs, if then. This would be millions of attempts.

Post reply on HN