Live data from Hacker News

How to break everything by fuzz testing

chameth.com

21–30 of 39 posts

Re: How to break everything by fuzz testing

#21
post #17
post #15

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

Fuzzing is not random testing though. It's directed random testing.

The original fuzzing was as random + black box as it gets.

Re: How to break everything by fuzz testing

#22
It can be difficult to evaluate the result of a test. We solved this by using an existing (of course inferior) library that uses a different algorithm for the same task (different algorithm so it fails at different tests). We would run the same test with both libraries and compare the results. If they were different, we had to find a way to decide which library failed or maybe evaluate those failed cases manually.

Re: How to break everything by fuzz testing

#23
post #9

> The fix for this was fairly straightforward - I just made the library keep a record of the previously visited IFDs and bail out if it found a loop. If you just want to detect loops, keep a “+1” pointer that you use to increment through the data; also, keep a “+2” pointer that is advanced twice each time your “+1” pointer advances: either your “+2” pointer hits the end, or it becomes equal to your “+1” pointer — in…

That's quite clever!

Re: How to break everything by fuzz testing

#24
post #9

> The fix for this was fairly straightforward - I just made the library keep a record of the previously visited IFDs and bail out if it found a loop. If you just want to detect loops, keep a “+1” pointer that you use to increment through the data; also, keep a “+2” pointer that is advanced twice each time your “+1” pointer advances: either your “+2” pointer hits the end, or it becomes equal to your “+1” pointer — in…

Also known as the "Tortoise and Hare Algorithm!"

https://en.wikipedia.org/wiki/Cycle_detection#Floyd's_Tortoi...

Re: How to break everything by fuzz testing

#25
post #5

Fuzzing is fun! If you're doing it on your personal computer (as opposed to a cloud VM somewhere), I'd suggest putting the testcase output directory on a spinning-rust hard drive that you don't care about instead of your (presumably much more expensive) internal SSD. It creates an impressive number of disk writes. I've been thinking about fuzzing JavaScript code (not attacking V8 or SpiderMonkey, but the JS code itse…

If the files you are fuzzing are small, then you could just create a couple of gigs of tmpfs ramdisk with something like "mount -t tmpfs -o size=2G none /mnt/somewhere" and put your fuzz directory on there.

Then the impressive number of writes are all to memory, which should pose no problem.

Re: How to break everything by fuzz testing

#26
post #15

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

You really shouldn't do random testing. Fuzzing is better, but property testing (fuzzing with shrinkage) is even better.

Re: How to break everything by fuzz testing

#27

Earlier quoted context omitted.

The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance. Would it? Maybe it's because I've had a "low-level upbringing", but whenever I'm writing parsing code for a file format, "assume any byte of data you read can have any value" is the norm. The rest of it follows from there.

Yeah, I've gotten to the point where I can't do any arithmetic on any values without immediately adding integer wrap tests afterwards.

Something the C# language gets right is that it can be configured to throw on overflow, or to wrap-around.

Re: How to break everything by fuzz testing

#29
post #8

My favorite personal fuzzing story is from 1987 when a friend said his x86 graphics drawing program was solid, and I said OK and smashed both hands on the keyboard and it insta-crashed.

Mark Twain put it best: The weakest of all weak things is a virtue which has not been tested in the fire.

Re: How to break everything by fuzz testing

#30
post #15

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

That piece of advice was probably valid in the 1970s: The computers were far too slow and far too expensive for any kind of random testing to make sense. Fuzzing became popular when multi-core CPUs became commonplace and RAM more affordable.
Post reply on HN