11–20 of 36 posts
link to the graphic for ease of reference:
https://blog.auxon.io/images/posts/effective-property-based-...
Auxon also looks like an interesting company.
[deleted]
Ask HN: Who is hiring? (February 2021)
Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confidence/safety of using property tests without introducing large delays into your CI pipeline?
If I were to add just one thing to the list: metatest. Write a test that asserts that your generated test cases are "sufficiently comprehensive", for whatever value of "sufficiently" you need. In an impure language, this is as easy as having the generator contain a mutable counter for "number of test cases meeting X condition" for whatever conditions you're interested in. For example, say your property is "A iff B". You might want to fail the test if fewer than 10% of the generated cases actually had A or B hold. (And then, of course, make sure your generators are such that - say - A and B hold 50% of the time; you want an astronomically small chance of random metatest failure.)
(I did a brief intro to this in the Metatesting section of a talk I did two years ago: https://github.com/Smaug123/talks/blob/master/DogeConf2019/D... . On rereading it now, I see there's a typo on the "bounded even integers" slide, where the final `someInts` should read `evenIntegers`.)
Property testing is awesome, but it does significantly slow down test suites. Are there any standard practices for striking a balance of getting the added confidence/safety of using property tests without introducing large delays into your CI pipeline?
We like to run the tests as part of CI with a relatively small number of iterations, and then turn the knob way up in a nightly or weekly scheduled test job.
The suggestion elsewhere in this thread to decrease the number of iterations during normal testing and crank it up during nightlies is also good.
The only thing I’m still missing from the libraries is a convenient mechanism of remembering previously failing generated inputs and use them as a static list of testcases next to the at runtime generated ones like a regression test of sorts.
Edit: typos
I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck ) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck ( https://www.scalacheck.org/ ). Both let you write your own generators and are quite good at shrinking down to minimal use cases. The suggestion e…
disclaimer: Auxon co-founder
Also, big (h/t) to Quickcheck from me as well. Getting into it via Erlang many, many years ago was among the more impactful and transformative developments in my approach to thinking about improving software quality.
I would like to make a shout out to quickcheck (Haskell, https://hackage.haskell.org/package/QuickCheck ) as it was my first introduction to property based testing and has since let me to use property based testing everywhere in the last 10 years. Then there is scalacheck ( https://www.scalacheck.org/ ). Both let you write your own generators and are quite good at shrinking down to minimal use cases. The suggestion e…
Indeed. It's an unfortunate general omission in the ecosystem. It's one of the things that our product does (in its appropriation from different inspirations like MC, PBT, SBFL, etc., but we move things up to assessing systems rather than programs) where contradictions and counter-examples are curated specifically for making it easier to reuse them. They're provided back to the user as executable "properties" in our…
In my job Cucumber seems to add little more than just commenting and sequencing functions, tasks that are better suited to your programming language of choice, while adding overhead and complexity.
What am I missing?
