A Year in the Life of a Compiler Fuzzing Campaign
blog.trailofbits.com
A Year in the Life of a Compiler Fuzzing Campaign
1–7 of 7 posts
Re: A Year in the Life of a Compiler Fuzzing Campaign
#2First time reading about compiler fuzzing, but seems like there's some parallels with TLA+.
With TLA+, you could check a compiler's specification before you implement it. Then, once you've implemented it, you could do fuzzing on the actual program.
I wonder how much overlap there is between bugs you could catch with TLA+ vs fuzzing.
Re: A Year in the Life of a Compiler Fuzzing Campaign
#3First time reading about compiler fuzzing, but seems like there's some parallels with TLA+. With TLA+, you could check a compiler's specification before you implement it. Then, once you've implemented it, you could do fuzzing on the actual program. I wonder how much overlap there is between bugs you could catch with TLA+ vs fuzzing.
The term you're looking for is model-based testing, where a formal description of the system is used to generate sequences of events which are fed as input to the system under test; the SUT state is then checked for equivalence to the model state. There is significant overlap in concept between model-based testing, property-based testing, and fuzzing: all involve the structured (some more, some less) generation of random inputs to a SUT and the checking of SUT behavior against an assumed-correct oracle. This spans from feeding random system files to the program to see whether it crashes with a strange exception, to a more formalized process directed by a TLA+ (or other language) spec as you've described.
Re: A Year in the Life of a Compiler Fuzzing Campaign
#4First time reading about compiler fuzzing, but seems like there's some parallels with TLA+. With TLA+, you could check a compiler's specification before you implement it. Then, once you've implemented it, you could do fuzzing on the actual program. I wonder how much overlap there is between bugs you could catch with TLA+ vs fuzzing.
The term you're looking for is model-based testing, where a formal description of the system is used to generate sequences of events which are fed as input to the system under test; the SUT state is then checked for equivalence to the model state. There is significant overlap in concept between model-based testing, property-based testing, and fuzzing: all involve the structured (some more, some less) generation of ra…
Thanks for naming "model-based testing"--added to my vocabulary. I was thinking more about this, and I would expect very little overlap in the bugs that you could expect to catch with model-based testing vs. fuzzing.
For example, if I get an error with TLA+--e.g. some state reaches deadlock, or there's an invariant that's violated by some behavior--it takes me a good deal of interpretation to see if there's actually a problem, or if just need to update my spec.
With fuzzing, it seems like the errors would be pretty clear to interpret. e.g. uncaught exceptions, or out-of-bounds memory accesses are clear problems with an implementation, and I would think takes less interpretation.
Re: A Year in the Life of a Compiler Fuzzing Campaign
#5> Did Solidity fuzzing run out of gas?
Heh.
Re: A Year in the Life of a Compiler Fuzzing Campaign
#6so far the only "language" i've fuzzed with AFL is an ANSI-BASIC dialect, and i used the first column from the token documentation to seed the dictionary.
A few words to the wise:
1) If you can fuzz in a VM or container with persistent storage, that's the way to do it. Fuzzing will absolutely hose your host file system (but not from a VM, hopefully.) It can't be "copy on X" sort of "delayed writes" either, because some fuzzing (sandsifter, for instance) will lock the host up. Even when fuzzing in a VM!
1.5) One way i've done it is to export iSCSI drives as the "work" drives and making the OS drive read only - but that was years ago back when "pivot_root" was something i was doing a dozen times a day on various machines.
2) AFL, the original that lcamtuf "finalized" several years ago, had no "out of the box" multicore fuzzing. But it did recognize "multiple fuzzers" in a directory with the M/S modes, and the actual "fuzzer input/output/results" directory could be on a network share. just make sure the "pwd" of the directory you run the fuzzer on the target binary/whatever in is not remotely near that network share. I suck at linux file systems so if there's a clever way to stop a misbehaving userspace binary from clobbering a filesystem, i'm all ears!
3) fuzzing can be very fun to mess around with even if you have no idea what the results mean, how to write a bug report, whatever. on lcamtuf's AFL page there was a demonstration of AFL building some graphics format from scratch, without a dictionary. I think the original seed input was "a\n" as ascii in the file, and that's it. Visualizing how the fuzzer was actually interacting with the file is cool. Seeing the BASIC programs the fuzzer would spit out after a CPU-month of fuzzing was interesting too...
Re: A Year in the Life of a Compiler Fuzzing Campaign
#7First time reading about compiler fuzzing, but seems like there's some parallels with TLA+. With TLA+, you could check a compiler's specification before you implement it. Then, once you've implemented it, you could do fuzzing on the actual program. I wonder how much overlap there is between bugs you could catch with TLA+ vs fuzzing.
Another approach to compiler fuzzing is blackbox differential fuzzing. Generate random valid programs and see if they do the same thing with different compiler settings, or even with two or more different compilers. This does mean you need to generate valid programs, which present challenges in languages with undefined or implementation-specific behaviors.
This is blackbox, so it doesn't allow one to start from nothing as a greybox fuzzer would.
Anyway, testing is not like programming in that different approaches are incompatible. Different testing approaches typically have different blind spots, so they complement each other.