Live data from Hacker News

How to break everything by fuzz testing

chameth.com

11–20 of 39 posts

Re: How to break everything by fuzz testing

#11

Interesting. Is there any fuzzing libraries for c#?

Can you call c# from C? If so, then you can just use any C fuzzing library, and have it call your C# code. I do this with C++ and Objective-C using clang's libfuzzer. You write a single C function that takes a pointer to a buffer and a length and pass it to whomever you want. I just write a C wrapper that calls my Objective-C or C++ functions with the data.

Re: How to break everything by fuzz testing

#12
post #2

My summary of this blog post: plenty of random input data can reveal code bugs. The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance.

The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance. Would it? Maybe it's because I've had a "low-level upbringing", but whenever I'm writing parsing code for a file format, "assume any byte of data you read can have any value" is the norm. The rest of it follows from there.

Yeah, I've gotten to the point where I can't do any arithmetic on any values without immediately adding integer wrap tests afterwards.

Re: How to break everything by fuzz testing

#13
post #2

My summary of this blog post: plenty of random input data can reveal code bugs. The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance.

The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance. Would it? Maybe it's because I've had a "low-level upbringing", but whenever I'm writing parsing code for a file format, "assume any byte of data you read can have any value" is the norm. The rest of it follows from there.

Let's go one step higher, keeping track of the state by a state machine. When designing/coding with the correctness on mind, I try to stay focused, and not think of edge cases. Or I will end up spending more time coming up with edge cases and what can go wrong. I'm not lazy, I'm almost certain of that. But I do feel time is a limited resource and want to add more value per hour spent working. Maybe, this is more the case of if it can be automated then automate it.

Re: How to break everything by fuzz testing

#14
post #2

My summary of this blog post: plenty of random input data can reveal code bugs. The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance.

The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance. Would it? Maybe it's because I've had a "low-level upbringing", but whenever I'm writing parsing code for a file format, "assume any byte of data you read can have any value" is the norm. The rest of it follows from there.

Although the really nasty bugs (like this one) happen when the individual bytes are all sensible, but the meta-level structure of the file is toxic.

Re: How to break everything by fuzz testing

#15
A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

Re: How to break everything by fuzz testing

#16
I have used fuzz testing to make my Bezier intersection library more robust against edge cases. The test would try to find all intersections between a random pair of curves that lie within certain bounds (you probably know that there can be up to nine intersections between two cubic Bezier curves). At the beginning it failed at approx. 1 in 100,000 randomly generated curve pairs, now I am at a point where there is not a single failure in a billion test.

My problem was how to decide if a test failed, because this would not be a crash, but failing to find an intersection between the curves. So I compared against an existing library which uses a completely different algorithm, which means the other library fails at other test cases than my own library. If the results in a test case were different, one must have failed and by testing against the found intersections I could easily decide which one.

Re: How to break everything by fuzz testing

#17
post #15

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

Fuzzing is not random testing though. It's directed random testing.

Re: How to break everything by fuzz testing

#18
post #5

Fuzzing is fun! If you're doing it on your personal computer (as opposed to a cloud VM somewhere), I'd suggest putting the testcase output directory on a spinning-rust hard drive that you don't care about instead of your (presumably much more expensive) internal SSD. It creates an impressive number of disk writes. I've been thinking about fuzzing JavaScript code (not attacking V8 or SpiderMonkey, but the JS code itse…

If you are interested in finding possible security holes, you could try finding prototype pollution bugs in basically any library that somehow handles user input. Utility libraries like lodash and underscore, argument parsers like yargs, minimist, others like moment, handlebars, DB/ODM tools like Mongoose, Knex, etc.

You'd look for code where input would be able to modify Object.prototype (or I guess some other constructor's prototype) unintentionally (and it's basically always unintentional).

Example of such vulnerability found in Minimist https://snyk.io/vuln/SNYK-JS-MINIMIST-559764

These issues are a constant pain in the JS ecosystem and you wouldn't be the only one using fuzzing to try to find them.

Re: How to break everything by fuzz testing

#20
post #19

In university we had to write a simple fuzzer which extracted options from man pages and ran the corresponding command with randomized but valid options. Didn't take long until we found the first bug in one of the tested commands.

that's a good assignment - extra marks for reporting the bugs?
Post reply on HN