Ask HN: What is the emerging state of the art in fuzzing techniques?

You might not be as far from the cutting edge as you'd expect.

From what I've seen, fuzzing is divided into two major camps (I'm generalizing to the extreme here):

1. Mutational - These include tools like AFL, are gaining traction in the open source community, and have a lot of applications, perhaps most notably in library and application fuzzing.

2. Generational - These include commercial tools Defensics and PeachFuzzer, and open source tools like Peach, Spike, and Sulley. The state of the art is held by commercial offerings in this camp, and it's what businesses are more likely to be interested in.

My hypothesis as to the reason for this split: Open source hackers are interested in finding bugs. Businesses are interested in assurance that their software is safe ("safe"). Protocol-specific tools give the impression that we've done the best we can at securing IP/TCP/TLS/HTTP/etc. Defensics is by far the dominant offering (in terms of apparent popularity), and Peach is the only active competitor I've ever found.

The open source generational branch is moving very slowly. The primetime candidate was once Peach, now called Peach Community [1]. Unfortunately the corporate backer switched to a closed solution, and left the open source tool out to dry. The latest tool of note besides Peach is Sulley [2] [3].

Books: I haven't found any books that go below the surface. "Fuzzing: Brute Force Vulnerability Discovery" has decent reviews on Amazon, but I found it more breadth than depth.

Papers:

1. IMO the seminal paper on fuzzing is Rauli Kaksonen's thesis, "A Functional Method for Assessing Protocol Implementation Security." [6] This will take you almost to the state of the practice. Kaksonen was a co-founder of Codenomicon. Very interesting read.

Talks: If you want cutting edge research, conference talks and blog posts may be as good as papers.

1. 2007 Blackhat conference Sulley talk "Fuzzing Sucks! - Introducing Sulley Fuzzing Framework" [2]

2. Google Charlie Miller fuzzing. My favorite slide decks are [7] and [8]. High fives (and a beverage on me should time and space ever permit) to anyone who can find audio or video from the actual talks.

Shameless plug(s):

1. Due to lack of response on Sulley pull requests, I forked to a new project called boofuzz [4], and I commit to at least address pull requests more quickly.

2. I'll be giving a fuzzing talk at Defcon 24's Packet Hacking Village which will address, among other things, the state of open source fuzzing [5].

[1]: http://www.peachfuzzer.com/resources/peachcommunity/

[2]: http://www.podcast.tv/video-episodes/pedram-amini-aaron-port...

[3]: https://github.com/OpenRCE/sulley

[4]: https://github.com/jtpereyda/boofuzz

[5]: https://www.wallofsheep.com/pages/dc24

[6]: http://www.vtt.fi/inf/pdf/publications/2001/P448.pdf

[7]: https://cansecwest.com/csw08/csw08-miller.pdf

[8]: http://pages.cs.wisc.edu/~rist/642-fall-2012/toorcon.pdf

For people as confused about this post as I was:

https://en.wikipedia.org/wiki/Fuzz_testing

in general, have a poke around https://fuzzing.info/papers/

First, I think the next big step in fuzzing will actually be a complement to fuzzing - solving.

AFL and friends can bitbang their way to massive code coverage, but can still fail on fairly simple testcases. Some recent research[1] by the authors of Angr[2] show that by pairing the brute-force coverage and exception discovery of a tool like AFL with constraint solving tools can really dig deep into a program, by actually solving the path to a given block of code. Microsoft's infamous SAGE fuzzer does this IIRC.

Second, I think there are still massive oportunities for fuzzing closed-source programs, as well as programs with tricky state, such as browsers or network daemons.

[1] https://www.internetsociety.org/sites/default/files/blogs-me...

[2] http://angr.io

Interestingly, Radamsa is implemented in Scheme, then transpiled to C. Originally I played around with invoke Radamsa as a server (it's normal usage model), but this wasn't ideal because I wanted to use it at varying granularities, which would have meant multiple invocations per mutate/execute cycle. What I ended up doing was to take the compiled Scheme, get rid of all the syscalls, link it directly into the fuzzer program binary, and turn its `main` function into something I could call directly!

This has been discussed to no end in the past. When I published my paper, I did so in a venue that had open-access to all its proceedings (HotPower 12). However CODASPY where this paper comes from isn't one of those venues. ACM explicitly denies permission to host the papers elsewhere - From the paper:

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

While we might question the ethics or morality of closed-access conference proceedings, it is definitely on the wrong side of legality. Much like patents are evil but can be defended/enforced by big corporations legally.

IMHO, professors and students who take public grants should publish only to open-access conferences and journals. If they do not do so, it does not justify illegally downloading the paper.