Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go Fuzz Testing

blog.fuzzbuzz.io

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Go Fuzz Testing

#11

Post by lalaithion »

evmunro wrote: post #6
lalaithion wrote: post #4

go test fuzz v1 string("000000000000000000000000000000Ö00000000000000000000000000000") rune('\u0083') int(60) Interesting that the minimization engine wasn't able to shrink this further.

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

For string inputs, some form of binary search ("Check if the bug exists in the first half or second half of the string") would be able to reduce this example to "Ö" in only a few iterations. Not sure if this just isn't implemented, or whether there's something more complex going on.

There's also the fact that I'd expect a fuzzer that knows about Unicode and UTF-8 strings to have a known list of weird behavior hardcoded as seed values, and certainly two-byte runes would be on that list.

Of course, this is only the first release with the fuzzer, and it already looks really amazing - all I'm really saying here is that I can't wait for these to be features of the fuzzer in the future!

Re: Go Fuzz Testing

#12

Post by andrei »

Thaxll wrote: post #8

Curious if fuzzing is a common thing in network programming, does fuzzing plays nicely with binary inputs?

It does! Fuzzing actually started off as a tool built by security researchers to find vulnerabilities in parsers, and other complex codebases, usually written in C/C++ (looking for memory bugs). So anything that deals with untrusted binary data is a prime candidate for fuzz testing.

Go’s fuzzing framework supports `[]byte` arguments as well as all of the standard Go primitives, so you should be able to test netcode this way.

If you're looking for a C/C++ solution, my recommendation is libfuzzer [0]. We've also built our own C/C++ fuzzing engine at Fuzzbuzz [1].

[0] https://llvm.org/docs/LibFuzzer.html

[1] https://docs.fuzzbuzz.io/docs/getting-started-in-c-or-c++

Re: Go Fuzz Testing

#13

Post by evmunro »

lalaithion wrote: post #11
evmunro wrote: post #6

Earlier quoted context omitted.

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

For string inputs, some form of binary search ("Check if the bug exists in the first half or second half of the string") would be able to reduce this example to "Ö" in only a few iterations. Not sure if this just isn't implemented, or whether there's something more complex going on. There's also the fact that I'd expect a fuzzer that knows about Unicode and UTF-8 strings to have a known list of weird behavior hardcod…

I agree - I took a look at the minimization algorithm[0] and it seems like it loops through a few basic options, with the last one basically normalizing all possible bytes to something readable (like "0"). Part of the issue with trying to be as generic as possible is you sometimes can't find the best solution to every problem, this might be one of those situations.

I know the goal of 1.18 was to get the UX down, so I'm interested to see how it improves for 1.19.

[0] https://github.com/golang/go/blob/master/src/internal/fuzz/m...

Re: Go Fuzz Testing

#14

Post by disposedtrolley »

lalaithion wrote: post #11
evmunro wrote: post #6

Earlier quoted context omitted.

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

For string inputs, some form of binary search ("Check if the bug exists in the first half or second half of the string") would be able to reduce this example to "Ö" in only a few iterations. Not sure if this just isn't implemented, or whether there's something more complex going on. There's also the fact that I'd expect a fuzzer that knows about Unicode and UTF-8 strings to have a known list of weird behavior hardcod…

You might be interested in reading about delta debugging: https://en.m.wikipedia.org/wiki/Delta_debugging

The algorithm has similar complexity as binary search, but is a bit smarter on deciding how to split the test input at each iteration.

I’ve been studying this in my masters, and we’ve recently had to write a Java implementation. I’m keen to start on a Go package soon that might work well with fuzz testing.

Re: Go Fuzz Testing

#15

Post by jiehong »

It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing.

I guess we still need to rely on something like gopter.

Re: Go Fuzz Testing

#16

Post by kardianos »

jiehong wrote: post #15

It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing. I guess we still need to rely on something like gopter.

Use table driven tests.

func TestFoo(t *testing.T) { list := []struct {

     want string
     input string
     expectedErr error
  }{
  {item1},
  {item2},
  }

  runtest

Re: Go Fuzz Testing

#19

Post by troutwine »

kardianos wrote: post #16
jiehong wrote: post #15

It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing. I guess we still need to rely on something like gopter.

Use table driven tests. func TestFoo(t *testing.T) { list := []struct { want string input string expectedErr error }{ {item1}, {item2}, } runtest

That’s not property testing. Yes there is a property being checked but you the user are selecting the inputs by hand which lumps in whatever biases you might bring to the test. Property testing is a property to validate, randomized search over the input space and, ideally, shrinking of inputs that fail the property. The benefit is you get to search over more of the input space that a human would normally do with an explicit table of inputs and with hopefully less bias toward inputs that don’t fail the property.

Re: Go Fuzz Testing

#20

Post by throwawaylala1 »

That first example is deceptive. In this line:

    for i := 0; i
The "When I did a quick scan of that code it looked good to me. Then he claimed it should panic and I felt crazy. So I copy/pasted the code into an editor to see wtf was going on and sure enough it was "<=".
Post reply on HN