Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the emerging state of the art in fuzzing techniques?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#31

Post by joosters »

wepple wrote: post #25

in general, have a poke around https://fuzzing.info/papers/ First, I think the next big step in fuzzing will actually be a complement to fuzzing - solving. AFL and friends can bitbang their way to massive code coverage, but can still fail on fairly simple testcases. Some recent research[1] by the authors of Angr[2] show that by pairing the brute-force coverage and exception discovery of a tool like AFL with constrain…

Has anyone tried combining fuzzing with 'all-pairs'/'pairwise' test case generation?

The idea of pairwise testing is that individual features in a program are commonly tested, but combinations of them are often poorly tested. However, trying to test all features with each other soon becomes a combinatorial nightmare. To deal with this, you use an algorithm (e.g. see the code for 'allpairs' at http://www.satisfice.com/tools.shtml ) to pick a minimal set of test cases that cover all possible pairs of configuration settings.

These test cases could then be used as starting points for fuzzing, to provide a greater code coverage faster.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#32

Post by Zardus »

pag wrote: post #26

I worked on the cyber reasoning system (CRS) at Trail of Bits for our entry into the Cyber Grand Challenge [1]. Some slides describing the system are here [2]. Specifically, I implemented our fuzzer. I created a dynamic binary translator [3] that emulated the DECREE [4] operating system and x86 arhcitecture. It had the Radamsa [5] mutator built-in, along with a number of other simpler mutators. I think our fuzzer out…

It's impressive how many resources you threw at problem! We (Shellphish) had very similar results by using AFL [4] for fuzzing and angr [5] for symbolic execution (we published our approach at NDSS in February [1]) on around 300 cores. Of course, we procrastinated pretty heavily, ended up hacking our CRS together in three weeks, and it was absurdly rough around the edges and didn't get anywhere near your crash numbers during the qualifying event itself. As we discuss in the paper, our experiments with the impressive numbers were carried out afterwards, in less chaotic conditions.

It's interesting that the approaches taken by us [1], you [2], and ForAllSecure [3] for the CQE (at least on the exploitation side) were so similar. I've talked with two other teams that had an analogous setup (as well as two other teams, who did quite well, that took a very different route). I guess some great minds think alike!

As a side note, in the ToB blog post, you talk about wanting to join up with another team to be able to play in the final event. Did you guys end up finding a partner? It'd be interesting to face your CRS again next month :-)

[1] https://www.internetsociety.org/sites/default/files/blogs-me... [2] https://blog.trailofbits.com/2015/07/15/how-we-fared-in-the-... [3] https://blog.forallsecure.com/2016/02/09/unleashing-mayhem/ [4] http://lcamtuf.coredump.cx/afl/ [5] http://angr.io

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#33

Post by roddux »

thornjm wrote: post #10
k__ wrote: post #5

Earlier quoted context omitted.

Really? I thought everyone dropped "security by obscurity" long time ago.

I was actually thinking along the lines of fuzzing for exploit development. You want a unique bug that will last a long time. In which case, your fuzzing techniques are a trade secret. A lot of fuzzing advances take place behind closed doors. Project Zero has some people with interesting backgrounds doing bug hunting for good.

I hold the belief that Google must be getting something else out of Project Zero other than just "we hire the best hackers" bragging rights.

I figure they're selling exploits (the ones they don't publicise) to governments.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#34

Post by pag »

Zardus wrote: post #32
pag wrote: post #26

I worked on the cyber reasoning system (CRS) at Trail of Bits for our entry into the Cyber Grand Challenge [1]. Some slides describing the system are here [2]. Specifically, I implemented our fuzzer. I created a dynamic binary translator [3] that emulated the DECREE [4] operating system and x86 arhcitecture. It had the Radamsa [5] mutator built-in, along with a number of other simpler mutators. I think our fuzzer out…

It's impressive how many resources you threw at problem! We (Shellphish) had very similar results by using AFL [4] for fuzzing and angr [5] for symbolic execution (we published our approach at NDSS in February [1]) on around 300 cores. Of course, we procrastinated pretty heavily, ended up hacking our CRS together in three weeks, and it was absurdly rough around the edges and didn't get anywhere near your crash number…

We ran three versions of our system, because in the last few weeks/days before the quals we had a bunch of regressions. One of those regressions was caused by a single line change in my fuzzer :-p On the twitter account we used to track our progress, we used rapper names for each version. One version was two weeks old, one was up-to-date with the fixed fuzzer, and one had an experimental version of one of our symbolic executors that produced tags. That didn't work.

We also under-utilized those nodes :-( Each node had at at least 4 idle cores wasting our money. Our resource allocation mechanism was naive.

I looked through some of the stuff released byt DARPA after the event and they released some of our PoVs as official PoVs. If you hex-decode them, you'll see something like "bad seed to Radamsa"!! That was a bug in how I would invoke Radamsa -- sometimes I'd pass it a seed that was way too big.

We tried to team up with every team but ForAllSecure. No one wanted to have our name on their ticket, or they were just fishing for details :-/ We've done a bit of work on the system since, getting it to work on Linux programs via a "port" of parts of libc to DECREE.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#35

Post by linschn »

zokier wrote: post #28
endergen wrote: post #14

Earlier quoted context omitted.

Why not?

Would you accompany a movie recommendation with a piratebay link on HN?

Except this research was (most probably, I did not check) funded with french or european taxpayers' money, and therefore people should not have to pay for it twice.

The moral standpoint is very different than with movie or music pirating.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#36

Post by mytummyhertz »

shameless self plug: https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...

allows you to run AFL on arbitrary VMs. so far we've used it to find some Linux vulnerabilities, and are starting to find stuff in other operating systems too. and we're just getting started :)

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#37

Post by yeukhon »

wepple wrote: post #25

in general, have a poke around https://fuzzing.info/papers/ First, I think the next big step in fuzzing will actually be a complement to fuzzing - solving. AFL and friends can bitbang their way to massive code coverage, but can still fail on fairly simple testcases. Some recent research[1] by the authors of Angr[2] show that by pairing the brute-force coverage and exception discovery of a tool like AFL with constrain…

I recommend https://www.usenix.org/search/site/fuzzing.

ACM library is fine as well, although I find more security-related journals in USENIX database.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#38

Post by nickpsecurity »

linschn wrote: post #35
zokier wrote: post #28

Earlier quoted context omitted.

Would you accompany a movie recommendation with a piratebay link on HN?

Except this research was (most probably, I did not check) funded with french or european taxpayers' money, and therefore people should not have to pay for it twice. The moral standpoint is very different than with movie or music pirating.

We should try to extend the First Sale Doctrine to digital publications. If it hasnt been tried already.

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#39

Post by Zardus »

pag wrote: post #34
Zardus wrote: post #32

Earlier quoted context omitted.

It's impressive how many resources you threw at problem! We (Shellphish) had very similar results by using AFL [4] for fuzzing and angr [5] for symbolic execution (we published our approach at NDSS in February [1]) on around 300 cores. Of course, we procrastinated pretty heavily, ended up hacking our CRS together in three weeks, and it was absurdly rough around the edges and didn't get anywhere near your crash number…

We ran three versions of our system, because in the last few weeks/days before the quals we had a bunch of regressions. One of those regressions was caused by a single line change in my fuzzer :-p On the twitter account we used to track our progress, we used rapper names for each version. One version was two weeks old, one was up-to-date with the fixed fuzzer, and one had an experimental version of one of our symboli…

You didn't approach us, either; bummer! I doubt we'd have been down to team up outright, but maybe some of you guys could have done an internship in our lab if you really wanted to work on the CGC? I guess it's ancient history now, though.

Resource utilization is definitely tricky. But man, the amount of resources you had is just mind-boggling! I just realized it's even more cores than DARPA gave us for the final event! I'm impressed you guys managed to keep it all running smoothly (at least, it seemed that way from our lab, where there was complete chaos as our system fell over and crapped itself repeatedly for the first few hours).

Re: Ask HN: What is the emerging state of the art in fuzzing techniques?

#40

Post by nickpsecurity »

pag wrote: post #27
pag wrote: post #26

I worked on the cyber reasoning system (CRS) at Trail of Bits for our entry into the Cyber Grand Challenge [1]. Some slides describing the system are here [2]. Specifically, I implemented our fuzzer. I created a dynamic binary translator [3] that emulated the DECREE [4] operating system and x86 arhcitecture. It had the Radamsa [5] mutator built-in, along with a number of other simpler mutators. I think our fuzzer out…

Interestingly, Radamsa is implemented in Scheme, then transpiled to C. Originally I played around with invoke Radamsa as a server (it's normal usage model), but this wasn't ideal because I wanted to use it at varying granularities, which would have meant multiple invocations per mutate/execute cycle. What I ended up doing was to take the compiled Scheme, get rid of all the syscalls, link it directly into the fuzzer p…

Badass tech and fuzzing results in your main comment. :) What Scheme or transpiler did you use, though?
Post reply on HN