Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day I Fell in Love with Fuzzing

nullprogram.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: The Day I Fell in Love with Fuzzing

#21

Post by UK-Al05 »

ergothus wrote: post #6

> When I got started, I had just learned how to use yacc (really Bison) and lex (really flex) I've dug into parsers a few times, but everything I encountered seemed to think that once I had a parsed tree of commands it was obvious how to consume it...and it wasn't (for me). I've never had the free time to dedicate to experimenting that abstractly, so anytime I'm tempted to write a DSL or similar for a current problem…

Tbh parser combinators and PEG parsers are the modern way of doing these sorts of things.

Re: The Day I Fell in Love with Fuzzing

#22

Post by evmunro »

Fuzzing is super powerful, but can be a bit complicated to set up - that's why I'm working on a fuzzing-as-a-service platform[1] that automates a bunch of the steps described here.

If you're interested in trying out fuzzing without having to learn the intricacies of AFL or set things up manually, let me know[2] and I can get you set up with an account to play around with.

Happy to answer any questions about AFL/Fuzzbuzz!

[1] https://fuzzbuzz.io

[2] everest [at] fuzzbuzz [dot] io

Re: The Day I Fell in Love with Fuzzing

#23

Post by Twirrim »

kragen wrote: post #17

I haven't tried afl-fuzz myself, although it sounds like world-class awesome software, but I'm a real believer in testing things with David MacIver's Hypothesis, which invokes your functions with random inputs, and then does similar canonicalization and minimization kinds of things. I like Hypothesis so much that when I wrote Dumpulse http://github.com/kragen/dumpulse I added a Python interface to it purely so I coul…

I've started using Hypothesis for testing my code when I'm working on Kattis problems, if I get a runtime exception of some form. It's especially valuable considering that the official execution environment is essentially a black box. I don't know the actual inputs that fail, and I don't know the exception raised to figure out what it is. In almost every case, I just have hypothesis be the test input and soon discover my bugs.

Re: The Day I Fell in Love with Fuzzing

#24

Post by felixyz »

sametmax wrote: post #20
kragen wrote: post #17

I haven't tried afl-fuzz myself, although it sounds like world-class awesome software, but I'm a real believer in testing things with David MacIver's Hypothesis, which invokes your functions with random inputs, and then does similar canonicalization and minimization kinds of things. I like Hypothesis so much that when I wrote Dumpulse http://github.com/kragen/dumpulse I added a Python interface to it purely so I coul…

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. unit tests are easy since I know what the code does, and I can just tell it what to do and what I expect. But with hypothesis, I have to find some kind of general property, which I have a hard time to do. Any tips, or materials I can use ? P.S: I read your test.py…

You might find these useful:

https://hypothesis.works/articles/rule-based-stateful-testin...

http://propertesting.com/book_stateful_properties.html

For a long time, I felt the same way you describe: all introductions to property-based testing show you things like testing a function that reverses a list and yes it all looks very useful and nifty, but it's hard to imagine how to formulate interesting variants for more complex code. These two posts were the first to really give me a fuller picture and lots of practical strategies:

https://fsharpforfunandprofit.com/posts/property-based-testi...

https://fsharpforfunandprofit.com/posts/property-based-testi...

(In general, everything I've ready/watched by Scott Wlaschin has been super helpful, and I've never written a line of F#.)

Re: The Day I Fell in Love with Fuzzing

#25

Post by sametmax »

felixyz wrote: post #24
sametmax wrote: post #20

Earlier quoted context omitted.

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. unit tests are easy since I know what the code does, and I can just tell it what to do and what I expect. But with hypothesis, I have to find some kind of general property, which I have a hard time to do. Any tips, or materials I can use ? P.S: I read your test.py…

You might find these useful: https://hypothesis.works/articles/rule-based-stateful-testin... http://propertesting.com/book_stateful_properties.html For a long time, I felt the same way you describe: all introductions to property-based testing show you things like testing a function that reverses a list and yes it all looks very useful and nifty, but it's hard to imagine how to formulate interesting variants for more…

Thanks a lot. This is why I still use HN :)

Re: The Day I Fell in Love with Fuzzing

#26

Post by felixyz »

sametmax wrote: post #25
felixyz wrote: post #24

Earlier quoted context omitted.

You might find these useful: https://hypothesis.works/articles/rule-based-stateful-testin... http://propertesting.com/book_stateful_properties.html For a long time, I felt the same way you describe: all introductions to property-based testing show you things like testing a function that reverses a list and yes it all looks very useful and nifty, but it's hard to imagine how to formulate interesting variants for more…

Thanks a lot. This is why I still use HN :)

This was my very first comment on HN. You made my day :)

Re: The Day I Fell in Love with Fuzzing

#27

Post by sametmax »

felixyz wrote: post #26
sametmax wrote: post #25

Earlier quoted context omitted.

Thanks a lot. This is why I still use HN :)

This was my very first comment on HN. You made my day :)

Just read it. Never heard of fsharpforfunandprofit before. Never touched f# either. But damn, that's good.

Re: The Day I Fell in Love with Fuzzing

#28

Post by rixrax »

evmunro wrote: post #22

Fuzzing is super powerful, but can be a bit complicated to set up - that's why I'm working on a fuzzing-as-a-service platform[1] that automates a bunch of the steps described here. If you're interested in trying out fuzzing without having to learn the intricacies of AFL or set things up manually, let me know[2] and I can get you set up with an account to play around with. Happy to answer any questions about AFL/Fuzzb…

What kind of closed source / commercial software can be fuzzed on your platform? E.g. how would you fuzz something like Adobe Lightroom or Autodesk AutoCAD there? What kind of reports would you provide?

Re: The Day I Fell in Love with Fuzzing

#29

Post by jononor »

sametmax wrote: post #20
kragen wrote: post #17

I haven't tried afl-fuzz myself, although it sounds like world-class awesome software, but I'm a real believer in testing things with David MacIver's Hypothesis, which invokes your functions with random inputs, and then does similar canonicalization and minimization kinds of things. I like Hypothesis so much that when I wrote Dumpulse http://github.com/kragen/dumpulse I added a Python interface to it purely so I coul…

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. unit tests are easy since I know what the code does, and I can just tell it what to do and what I expect. But with hypothesis, I have to find some kind of general property, which I have a hard time to do. Any tips, or materials I can use ? P.S: I read your test.py…

How I often approach this is to 1) write a couple of unit tests 2) refactor unit tests into data-driven tests, using pytest.mark.parametrize 3) use hypothesis to generate the data. I very often get to 2, but not always to 3.

The most basic things to check

- No valid inputs give an error

- All invalid inputs give a error

These are enhanced if you sprinkle your code with assertions. Writing assertions inside the code is good training to identify. Something supposed always non-negative, check it. Something always supposed to have more than 0 entries, check it. Something supposed to be sorted, check a[0] Often you can test symmetry type properties

- deserialize(serialize(in)) == in.

- init(); connect(); disconnect(); -> state identical as just init();

Or you can use an 'oracle', something external that tells you whether the output is correct or not.

- allTricksOptimized(in) == simpleReferenceImplementation(in)

- myImplementation(in) == thirdparty.implementation(in)

Re: The Day I Fell in Love with Fuzzing

#30

Post by sametmax »

jononor wrote: post #29
sametmax wrote: post #20

Earlier quoted context omitted.

I love the concept of hypothesis, but I struggle with finding a real life use case. I just can't formulate my assertions, I don't know what to put in them. unit tests are easy since I know what the code does, and I can just tell it what to do and what I expect. But with hypothesis, I have to find some kind of general property, which I have a hard time to do. Any tips, or materials I can use ? P.S: I read your test.py…

How I often approach this is to 1) write a couple of unit tests 2) refactor unit tests into data-driven tests, using pytest.mark.parametrize 3) use hypothesis to generate the data. I very often get to 2, but not always to 3. The most basic things to check - No valid inputs give an error - All invalid inputs give a error These are enhanced if you sprinkle your code with assertions. Writing assertions inside the code i…

Thanks. felixyz links did contains all those, but I'll remember the trick about pytest.mark.parametrize as a step toward hypothesis.
Post reply on HN