Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effective Property-Based Testing

blog.auxon.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Effective Property-Based Testing

#21

Post by mullr »

spockz wrote: post #19
im_down_w_otp wrote: post #17

Earlier quoted context omitted.

Indeed. It's an unfortunate general omission in the ecosystem. It's one of the things that our product does (in its appropriation from different inspirations like MC, PBT, SBFL, etc., but we move things up to assessing systems rather than programs) where contradictions and counter-examples are curated specifically for making it easier to reuse them. They're provided back to the user as executable "properties" in our…

In the article you (auxon) wrote to avoid using type based generators and unbounded collections and instead write your own generators. Usually I find it more convenient to write filters on the generated types then create my own generators. As the post mentions, this can lead to many discarded inputs. Have you ever considered how we could use the predicates in the filters to create Specialized generators that only gen…

> Have you ever considered how we could use the predicates in the filters to create Specialized generators that only generate inputs that will be accepted by the filter? You probably need some meta programming or reification of the predicates for that to work.

That's a really nice idea. It doesn't fit into the usual compositional design of the proptest libraries that I've used, but it certainly seems like it should be possible in principle.

Re: Effective Property-Based Testing

#22

Post by mullr »

lowercase1 wrote: post #20

This is a good overview of property based testing but the mentions of Cucumber threw me off. In my job Cucumber seems to add little more than just commenting and sequencing functions, tasks that are better suited to your programming language of choice, while adding overhead and complexity. What am I missing?

To me, Cucumber is a way to write executable specifications that can also be read by non-developers. This can be tremendously powerful if done judiciously, or tremendously pointless if not. This may be because you do not in fact want or need executable specification, or perhaps because there's nobody interested in reading it.

I believe that Cucumber is at its best in situations where it's clear to all parties that a specification is valuable. In that case, making the specification executable is very clearly a massively useful way to spend your time.

Re: Effective Property-Based Testing

#24

Post by dwohnitmok »

spockz wrote: post #16

I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck ) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck ( https://www.scalacheck.org/ ). Both let you write your own generators and are quite good at shrinking down to minimal use cases. The suggestion e…

> I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck) as it was my first introduction to property based testing

As far as I know the Haskell Quickcheck library by John Hughes was in fact the first library to lay out property-based testing. He then went on to create a paid and expanded version of the library in Erlang. And then as Quickcheck rose in popularity it's been re-implemented in many many different languages.

Re: Effective Property-Based Testing

#25

Post by pfdietz »

Turning off shrinking makes little sense to me. If tests are passing there's nothing to shrink. If you are bombarded with so many failures you can't afford to shrink them then more testing is pointless.

Re: Effective Property-Based Testing

#27

Post by amw-zero »

spockz wrote: post #16

I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck ) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck ( https://www.scalacheck.org/ ). Both let you write your own generators and are quite good at shrinking down to minimal use cases. The suggestion e…

Hypothesis does what you’re talking about. It stores failing examples and chooses them during subsequent runs.

Re: Effective Property-Based Testing

#28

Post by amw-zero »

pfdietz wrote: post #25

Turning off shrinking makes little sense to me. If tests are passing there's nothing to shrink. If you are bombarded with so many failures you can't afford to shrink them then more testing is pointless.

The author’s point is that shrinking can take a very long time, and that’s what you should consider turning it off. In my experience this is also true.

They’re saying that sometimes it’s better to just see the error in full and try and figure it out.

Re: Effective Property-Based Testing

#29

Post by amw-zero »

k__ wrote: post #7
the_gastropod wrote: post #5

Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confidence/safety of using property tests without introducing large delays into your CI pipeline?

You could fan-out tests to multiple FaaS instances. Cloudflare Workers, for example, have no cold-starts. But they are JS/WASM only

That’s a lot of buzzwords you got there

Re: Effective Property-Based Testing

#30

Post by mristin »

spockz wrote: post #19
im_down_w_otp wrote: post #17

Earlier quoted context omitted.

Indeed. It's an unfortunate general omission in the ecosystem. It's one of the things that our product does (in its appropriation from different inspirations like MC, PBT, SBFL, etc., but we move things up to assessing systems rather than programs) where contradictions and counter-examples are curated specifically for making it easier to reuse them. They're provided back to the user as executable "properties" in our…

In the article you (auxon) wrote to avoid using type based generators and unbounded collections and instead write your own generators. Usually I find it more convenient to write filters on the generated types then create my own generators. As the post mentions, this can lead to many discarded inputs. Have you ever considered how we could use the predicates in the filters to create Specialized generators that only gen…

You can already do that with a family of predicates if you write preconditions in your Python code (see my previous comment [1]). There is an ongoing discussion how to bring this in into Hypothesis (see the issue [2]).

[1]: https://news.ycombinator.com/item?id=26018386

[2]: https://github.com/HypothesisWorks/hypothesis/issues/2701

edit: newlines

Post reply on HN