Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to break everything by fuzz testing

chameth.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: How to break everything by fuzz testing

#31

Post by pfdietz »

dnautics wrote: post #26
pfdietz wrote: post #15

A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.

You really shouldn't do random testing. Fuzzing is better, but property testing (fuzzing with shrinkage) is even better.

The original prejudice was against any sort of randomness in testing. Manually constructed tests were seen as superior. That may have been true when computer time was dear, but the bias persisted into the latest edition of a well known book on software testing, published after (for example) Csmith had been released.

Re: How to break everything by fuzz testing

#33

Post by Psyladine »

"A QA engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd.

First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone."

Re: How to break everything by fuzz testing

#34

Post by UncleMeat »

pfdietz wrote: post #21
praptak wrote: post #17

Earlier quoted context omitted.

Fuzzing is not random testing though. It's directed random testing.

The original fuzzing was as random + black box as it gets.

Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.

Re: How to break everything by fuzz testing

#35

Post by pfdietz »

UncleMeat wrote: post #34
pfdietz wrote: post #21

Earlier quoted context omitted.

The original fuzzing was as random + black box as it gets.

Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.

Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.

Re: How to break everything by fuzz testing

#36

Post by UncleMeat »

pfdietz wrote: post #35
UncleMeat wrote: post #34

Earlier quoted context omitted.

Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.

Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.

That's actually one of the few fields where I feel like fuzzing has underperformed. There was an interesting paper at OOPSLA this year that found that while the fuzzing community has indeed found a lot of bugs, that these bugs actually are triggered by real code approximately-never. It was a really interesting result coming from within a community that ordinarily biases towards overinflating the value of PL techniques.

Re: How to break everything by fuzz testing

#37

Post by pfdietz »

UncleMeat wrote: post #36
pfdietz wrote: post #35

Earlier quoted context omitted.

Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.

That's actually one of the few fields where I feel like fuzzing has underperformed. There was an interesting paper at OOPSLA this year that found that while the fuzzing community has indeed found a lot of bugs, that these bugs actually are triggered by real code approximately-never. It was a really interesting result coming from within a community that ordinarily biases towards overinflating the value of PL technique…

That paper, if it's the one I'm thinking of, found that a compiler bug found by fuzzing was more likely to be found by a user, than a user found bug was likely to be found by another user. So if fuzzing-found bug reports are bad, user-found bug reports are even less useful.

Another thing to remember is that as blackbox fuzzing became state of the practice, its benefit declined, as the bugs it would find would be found early, by the developers themselves. All testing techniques are self-limiting this way.

I want you to look at the results of jsfunfuzz and tell me the impact of that wasn't profound.

Re: How to break everything by fuzz testing

#38

Post by snazz »

thewebcount wrote: post #11
ToFab123 wrote: post #10

Interesting. Is there any fuzzing libraries for c#?

Can you call c# from C? If so, then you can just use any C fuzzing library, and have it call your C# code. I do this with C++ and Objective-C using clang's libfuzzer. You write a single C function that takes a pointer to a buffer and a length and pass it to whomever you want. I just write a C wrapper that calls my Objective-C or C++ functions with the data.

Doesn't libFuzzer only require `extern "C" int LLVMFuzzerTestOneInput(...` to fuzz C++ code? What else does your C wrapper do beyond that? Google puts their fuzz tests right alongside the rest of the Chromium source code, which is C++.

Re: How to break everything by fuzz testing

#39

Post by mcswell »

jra_samba_org wrote: post #12
userbinator wrote: post #3

Earlier quoted context omitted.

The kind of bugs that would take probably a lot of time of think of and write unit tests for, in advance. Would it? Maybe it's because I've had a "low-level upbringing", but whenever I'm writing parsing code for a file format, "assume any byte of data you read can have any value" is the norm. The rest of it follows from there.

Yeah, I've gotten to the point where I can't do any arithmetic on any values without immediately adding integer wrap tests afterwards.

Reminds me of using a slide rule. You normally push the inner part (the C scale) to the right, line up the 1 on the C scale with the first number you're multiplying on the D scale, then look on the C scale for the second number you're multiplying, and read the result off the D scale immediately below that.

But when the result is more than 10, you've wrapped: your answer is off the D scale. So now you have to push the inner part back to the left, and line up the 10 (usually marked as 1, at the right-hand end) on the C scale with the first number on the D scale. And remember to add 1 to the exponent.

I've seen slide rules where the D scale goes slightly beyond 10 (like 10.1), so if the result was just a tiny bit over 10, you wouldn't need to wrap.

Post reply on HN