Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fuzz testing: the best thing to happen to our application tests

questdb.io

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Fuzz testing: the best thing to happen to our application tests

#21

Post by rwmj »

yosefk wrote: post #2

The relative rarity of input (pseudo-)randomization in SW testing is near inexplicable to me, except by the very low cost of all but the most commonly reproducing bugs paid by the SW vendor.

I love fuzzing as a technique and use it quite regularly and I'm even the maintainer of AFL++ in Fedora. But running AFL++ on even a single program occupies all threads of a high end AMD server for weeks. I'm running it locally so merely paying for the electricity. If it was a cloud instance it would cost a small fortune. I think this is a reason it is not used more widely. In addition most CI systems assume the tests will run in a small finite amount of time, not run for weeks on end.

I will note that Google have a programme for doing fuzz testing on open source projects using compute from their cloud: https://google.github.io/oss-fuzz/

Re: Fuzz testing: the best thing to happen to our application tests

#22

Post by rwmj »

todd8 wrote: post #9

I was working on Distributed Services for AIX in 1986 and 1987, a distributed filesystem to compete with NFS. As this was being developed by a dev team, my colleague and I pondered how to test the system that we had architected. There were so many possible states that a system's file system can be in. Were the conventional tests going to catch subtle bugs? Here's an example of an unusual but not unheard of issue: in…

The key to modern fuzzing is feedback, usually some kind of coverage measurement of the program under test. This allows the fuzzer to be much smarter about how it finds new code paths and discards inputs that don't extend coverage. This makes fuzzing find bugs a lot quicker.

Google have a project to do fuzzing on Linux system calls using coverage feedback: https://github.com/google/syzkaller

Re: Fuzz testing: the best thing to happen to our application tests

#23

Post by rwmj »

jacquesm wrote: post #13
mqus wrote: post #5

Earlier quoted context omitted.

In the regular testsuite (think CI) you want to have predictable results. Doing them again and again on the same code should give the same results so you can properly see with which code change things got wrong. Maybe it's simpler to explain it the other way around, for every new path your fuzzer(or other randomized test) tests, it also doesn't test a path it tested in a previous run and you probably want to add the…

Any half decent fuzzing setup will log what it did prior so you can replay it to the point of failure. This gets a lot harder when you do multiple such runs in parallel.

AFL++ logs the specific input that causes the crash. In theory at least replaying the input ought to trigger the crash reproducibly. (Sometimes not the case if the program has lots of threads or is event driven or otherwise stochastic).

Re: Fuzz testing: the best thing to happen to our application tests

#24

Post by jacquesm »

farresito wrote: post #20
jacquesm wrote: post #11

Earlier quoted context omitted.

That's very much my experience: asymptotic reduction in bug incidence and matching increase in runtime between subsequent errors. To the point that they are so rare that you think you have fixed all of them. But that's usually an illusion: it's just that the error rate is now so low that you no longer observe incidents yourself. The only way to get past that hurdle is to do the bad thing: release and hope that you go…

> The better way is to design it from the ground up with a bunch of instrumentation that keeps all of your invariants under close observation and that stops the moment anything is not according to your assumptions. Any personal "war stories" you are willing to share explaining how you went about designing such a system? :-) Or any presentation of someone who did that?

I wrote about retrofitting such a system onto an existing one:

https://jacquesmattheij.com/all-programming-is-bookkeeping/

Which was written in the context of:

https://jacquesmattheij.com/saving-a-project-and-a-company/

And without which that project likely would not have seen a successful end result.

It really is a war story, and what I like most about it is how at the drop of a hat a team assembled to get the job done, cleanly tackled the problem(s) and transferred duties to a new crew.

Re: Fuzz testing: the best thing to happen to our application tests

#26

Post by vodou »

(Sorry for hijacking the thread with a general fuzz question.)

I want to do fuzz testing on a library/framework written in C++. The actual target to test against is a simulator that takes input from a socket according to a network protocol. This simulator is built on both Linux and Windows.

What fuzzing frameworks would you recommend for this? There are quite a few and not always easy for me (as a fuzzing beginner) to understand the differences between them. Some of them seems to be abandoned projects as well.

Re: Fuzz testing: the best thing to happen to our application tests

#27

Post by Uw7yTcf36gTc »

5440 wrote: post #10

For those of you in FDA regulated devices, my clients started receiving FDA NSE letters for not performing fuzz testing. For example, "Though you have provided penetration testing, it does not appear that you have addressed the other items identified such as static and dynamic code analysis, malformed input (fuzz) testing, or vulnerability scanning. This testing is necessary to assess the effectiveness of the cyberse…

If their penetration testing didn’t perform fuzzing then you may want to look into a new pen test provider. Fuzz testing is default on most pen tests (I do this professionally)
Post reply on HN