81–90 of 90 posts
Earlier quoted context omitted.
It's unrelated to this article, but I suppose this motivation is why no one has a Gremlins-like feature any more. It seems to be almost totally forgotten, since the only link I could find is an excerpt from a PalmOS programming book: https://www.oreilly.com/library/view/palm-programming-the/15...
The gremlins turned into monkeys (as they quickly did in the page you shared as well): https://developer.android.com/studio/test/other-testing-tool...
Unfortunately it ends before it gets to the good stuff. It has me interested that maybe PBT can find some bugs that unit testing wouldn't - however I'm not sure how to write a PBT that would catch those bugs. The obvious tests that drive PBT advocates to drink are not interesting - unit tests will catch all the errors and because there is no randomness they will catch the errors faster in general. However how do I wr…
- "Metamorphic testing" is where analyze how code changes with changing inputs. For example, adding more filters to a query should return a strict subset of the results, or if a computer vision system recognizes a person, it should recognize the same person if you tilt the image.
- Creating a simplified model of the code, and then comparing the code implementation to the model, a la https://matklad.github.io/2024/07/05/properly-testing-concur... or https://johanneslink.net/model-based-testing
There's also this paper, which I haven't read yet but seems intriguing: https://andrewhead.info/assets/pdf/pbt-in-practice.pdf
Earlier quoted context omitted.
It's unrelated to this article, but I suppose this motivation is why no one has a Gremlins-like feature any more. It seems to be almost totally forgotten, since the only link I could find is an excerpt from a PalmOS programming book: https://www.oreilly.com/library/view/palm-programming-the/15...
That sounds like fuzz testing; which is similar to PBT, but (a) usually checks a single property ("the program doesn't crash") and (b) sends data via the program/system's ordinary input channels (whereas PBT has "white box" access to internals, like unit test do).
In bigger programs this is an outright necessity because pure random fuzzing would basically be a lottery.
I've always felt that unit testing frameworks and libraries and even parameterized testing were missing this kind of functionality.
Intellij is able to run my tests and figure out the code coverage, but why isn't it closing the loop and auto-fuzzing/auto-discovering how to mutate tests to cover more?
And don't point me at AI, none of this requires AI and nothing should have to "think" to do this.
It's crazy to me that the vast majority of code running all the time is not exhaustively tested through almost all of it's possible state space with most of it's possible input space. It's not like we are lacking the CPU bandwidth to do it.
Why can't I write a new function and have something tell me within ten minutes "this input param causes an exception" without any effort from me? Instead all those extra cores in my CPU just run javascript trash and crowdstrike scanners
Earlier quoted context omitted.
Your last paragraph is using terms like "correct", "fix" and "bug" as if they're absolute, when they're actually relative to some sort of spec (whether formal or informal, written or vibes-based, etc.). If the organisation controls the spec, then it can be perfectly reasonable for them to update that spec to e.g. allow certain behaviours that previously would have been considered bugs. In that case, we update the pro…
The spec becomes "don't break important customers code". How could one possibly formalize that?
Earlier quoted context omitted.
The spec becomes "don't break important customers code". How could one possibly formalize that?
The same way one formalises the implementation (AKA writing code): iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc.
What's more PBT doesn't depend on having a spec, just on having some properties that hold. So you very possibly didn't have a spec to start with.
Earlier quoted context omitted.
That sounds like fuzz testing; which is similar to PBT, but (a) usually checks a single property ("the program doesn't crash") and (b) sends data via the program/system's ordinary input channels (whereas PBT has "white box" access to internals, like unit test do).
Good fuzzing workflows and tools use the flow of code and which branches get taken to help find correlations between inputs and outcomes, and uses that extra context to more efficiently fuzz inputs. In bigger programs this is an outright necessity because pure random fuzzing would basically be a lottery. I've always felt that unit testing frameworks and libraries and even parameterized testing were missing this kind…
Sure, but that's an optimisation/implementation-detail. Similar to how PBT frameworks tend to use random generation + shrinking: it's not fundamental to the approach, but turns out to be much more effective than e.g. enumerative testing (e.g. Smallcheck), or showing un-shrunk examples.
> I've always felt that unit testing frameworks and libraries and even parameterized testing were missing this kind of functionality.
Coverage-guided PBT seems to have been re-invented several times (e.g. using QuickCheck with HPC in Haskell), though all the examples I've seen are toys or experiments. Hypothesis has experimental support for generating data using an external fuzzer, which presumably uses coverage (though I've not tried that feature yet).
> Why can't I write a new function and have something tell me within ten minutes "this input param causes an exception" without any effort from me?
I agree. One piece of advice is to respect the options provided by PBT frameworks, e.g. for the number of tests to run, the maximum discard:success ratio, the maximum "size" to pass into generators, etc. These can be tweaked per property, e.g. if a particular property is slowing down our CI we might want to only test it 20 times instead of the default of 100. However, I always make sure to transform that default value (in this case dividing it by 5) rather than setting a particular number: that way the test suite can also be run with bigger options to get a more thorough search (e.g. locally in the background, or by another CI job that's run less often, etc.).
Unfortunately I've not come across a framework that will keep on checking properties continuously (say, in a round-robin fashion). Sticking the test command in a loop should probably work though: `while runTests; do sleep 1; done; notify "Tests failed!"`
PS: As for "AI", I think it's better to be more specific. LLMs certainly aren't needed for this; but fuzzers have been using GOFAI techniques like genetic algorithms for a long time!
Earlier quoted context omitted.
The same way one formalises the implementation (AKA writing code): iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc.
That makes no sense. How do you know what bugs the customer depends on, especially if you don't have access to their code? What's more PBT doesn't depend on having a spec, just on having some properties that hold. So you very possibly didn't have a spec to start with.
You don't know, but you can ask, you can make educated guesses, etc. As I said, we do this stuff iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc. That's what most programming is.
> What's more PBT doesn't depend on having a spec, just on having some properties that hold
I'd say that "having some properties that hold" certainly counts as "some sort of spec (whether formal or informal, written or vibes-based, etc.)".
> So you very possibly didn't have a spec to start with.
There's always "some sort of spec"; even if it starts as vague as "let's try to make some money using computers".
Earlier quoted context omitted.
For code that's more "business logic" rather than "algorithmic", I find the following helpful: - Despite the terrible tutorial examples, PBT isn't about running one function on an arbitrary input, then trying to think of assertions about the result. Instead, focus on ways that different parts of your production code fits together, what assumptions are being made at each point, etc. - You don't need to plug random inp…
https://hypothesis.readthedocs.io/en/latest/stateful.html That last test style you describe can be done with Hypothesis. I've had some good success testing both Python programs and programs written in other languages that could be driven from Python with it. Like a server using gRPC (or CORBA once) as an interface, driven by tests written in Python imitating client behavior.
(For context, some "stateful" things that I've tested using ordinary PBT include browser automation (Hypothesis + Chrome + ChromeDriver), window manager scripts (QuickCheck + polysemy + Yabai), and a dynamic binding implementation for the JVM (ScalaCheck + a bespoke DSL for testing that it works correctly with Futures))
There's a good discussion of stateful property testing at https://stevana.github.io/the_sad_state_of_property-based_te... but personally I'm more interested in the parallel/concurrent/linearisability aspect that's also discussed.
Earlier quoted context omitted.
Nah. That's not what this is about. In any system worth testing, with property-based testing, the system will take so many steps before it encounters an error that even knowing what the steps were isn't going to be very helpful in reproducing the error. I've been there. It takes many hours to try to guess where the system went wrong to produce the undesirable result, and then you still might not be sure if you are lo…
Yes, timeouts are bad, but that’s true even for regular tests with determininistic input. You’ll get flakes running on different machines, depending on how much load is on them. To test an entire system with reproducable failures, you probably need something more heavyweight like Antithesis. Property tests are more useful for unit tests.
But, in general, yes, unit test failure with many steps would've been just as difficult to interpret.
My experience though was that once such a difficult failure is encountered during property-based testing, one has to write a unit test to reproduce the error anyways. But it's hard to assess the probability of the unexpected behavior of being an actual bug. Sometimes you discover that the system behavior was underspecified, or that you misunderstood how the system is supposed to behave after reading the specification.
Aw man, I was nodding along with the "most examples suck" section and then... it ended :(
Here's a great talk by John Hughes, one of the authors of QuickCheck, with real-life examples: https://www.youtube.com/watch?v=zi0rHwfiX1Q