Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fuzz testing: the best thing to happen to our application tests

questdb.io

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Fuzz testing: the best thing to happen to our application tests

#11

Post by jacquesm »

todd8 wrote: post #9

I was working on Distributed Services for AIX in 1986 and 1987, a distributed filesystem to compete with NFS. As this was being developed by a dev team, my colleague and I pondered how to test the system that we had architected. There were so many possible states that a system's file system can be in. Were the conventional tests going to catch subtle bugs? Here's an example of an unusual but not unheard of issue: in…

That's very much my experience: asymptotic reduction in bug incidence and matching increase in runtime between subsequent errors. To the point that they are so rare that you think you have fixed all of them. But that's usually an illusion: it's just that the error rate is now so low that you no longer observe incidents yourself. The only way to get past that hurdle is to do the bad thing: release and hope that you got it right and if you didn't that the incidents will not be too bad.

You could run many tests in parallel to reduce the chance but it will never be completely zero. Writing bug free software this way is hard. The better way is to design it from the ground up with a bunch of instrumentation that keeps all of your invariants under close observation and that stops the moment anything is not according to your assumptions. This usually gets you to a high level of confidence that things really do work as designed. But of course, that also isn't perfect and residual risk (and residual bugs...) will always remain in any system of even moderate complexity. File systems are well above that level, especially distributed file systems.

Re: Fuzz testing: the best thing to happen to our application tests

#12

Post by jacquesm »

5440 wrote: post #10

For those of you in FDA regulated devices, my clients started receiving FDA NSE letters for not performing fuzz testing. For example, "Though you have provided penetration testing, it does not appear that you have addressed the other items identified such as static and dynamic code analysis, malformed input (fuzz) testing, or vulnerability scanning. This testing is necessary to assess the effectiveness of the cyberse…

That's excellent that they are doing that. Especially for embedded devices because there tend to be lots of homebrew protocols on those, and those are usually easy pickings.

Re: Fuzz testing: the best thing to happen to our application tests

#13

Post by jacquesm »

mqus wrote: post #5
yosefk wrote: post #2

The relative rarity of input (pseudo-)randomization in SW testing is near inexplicable to me, except by the very low cost of all but the most commonly reproducing bugs paid by the SW vendor.

In the regular testsuite (think CI) you want to have predictable results. Doing them again and again on the same code should give the same results so you can properly see with which code change things got wrong. Maybe it's simpler to explain it the other way around, for every new path your fuzzer(or other randomized test) tests, it also doesn't test a path it tested in a previous run and you probably want to add the…

Any half decent fuzzing setup will log what it did prior so you can replay it to the point of failure. This gets a lot harder when you do multiple such runs in parallel.

Re: Fuzz testing: the best thing to happen to our application tests

#14

Post by aaron695 »

mqus wrote: post #5
yosefk wrote: post #2

The relative rarity of input (pseudo-)randomization in SW testing is near inexplicable to me, except by the very low cost of all but the most commonly reproducing bugs paid by the SW vendor.

In the regular testsuite (think CI) you want to have predictable results. Doing them again and again on the same code should give the same results so you can properly see with which code change things got wrong. Maybe it's simpler to explain it the other way around, for every new path your fuzzer(or other randomized test) tests, it also doesn't test a path it tested in a previous run and you probably want to add the…

[dead]

Re: Fuzz testing: the best thing to happen to our application tests

#15

Post by code_biologist »

jacquesm wrote: post #11
todd8 wrote: post #9

I was working on Distributed Services for AIX in 1986 and 1987, a distributed filesystem to compete with NFS. As this was being developed by a dev team, my colleague and I pondered how to test the system that we had architected. There were so many possible states that a system's file system can be in. Were the conventional tests going to catch subtle bugs? Here's an example of an unusual but not unheard of issue: in…

That's very much my experience: asymptotic reduction in bug incidence and matching increase in runtime between subsequent errors. To the point that they are so rare that you think you have fixed all of them. But that's usually an illusion: it's just that the error rate is now so low that you no longer observe incidents yourself. The only way to get past that hurdle is to do the bad thing: release and hope that you go…

I think there's room for a hybrid approach, not just fixing the bug a fuzz test found, but treating the fuzz test error as a bug in invariants (either inadequate or violated in a subtle way that's not caught) and addressing that underlying issue.

Re: Fuzz testing: the best thing to happen to our application tests

#16

Post by jacquesm »

code_biologist wrote: post #15
jacquesm wrote: post #11

Earlier quoted context omitted.

That's very much my experience: asymptotic reduction in bug incidence and matching increase in runtime between subsequent errors. To the point that they are so rare that you think you have fixed all of them. But that's usually an illusion: it's just that the error rate is now so low that you no longer observe incidents yourself. The only way to get past that hurdle is to do the bad thing: release and hope that you go…

I think there's room for a hybrid approach, not just fixing the bug a fuzz test found, but treating the fuzz test error as a bug in invariants (either inadequate or violated in a subtle way that's not caught) and addressing that underlying issue.

Yes, good point: often it is the assumptions themselves that were broken. So effectively you are debugging both the real system and the monitoring system.

Re: Fuzz testing: the best thing to happen to our application tests

#17

Post by kragen »

yosefk wrote: post #2

The relative rarity of input (pseudo-)randomization in SW testing is near inexplicable to me, except by the very low cost of all but the most commonly reproducing bugs paid by the SW vendor.

hardware people keep saying that for some reason

maybe someday software people will listen

that would be a good day

Re: Fuzz testing: the best thing to happen to our application tests

#18

Post by kragen »

mqus wrote: post #5
yosefk wrote: post #2

The relative rarity of input (pseudo-)randomization in SW testing is near inexplicable to me, except by the very low cost of all but the most commonly reproducing bugs paid by the SW vendor.

In the regular testsuite (think CI) you want to have predictable results. Doing them again and again on the same code should give the same results so you can properly see with which code change things got wrong. Maybe it's simpler to explain it the other way around, for every new path your fuzzer(or other randomized test) tests, it also doesn't test a path it tested in a previous run and you probably want to add the…

hypothesis kind of solves this problem by adding each (minimized) failing input to a file and always running it thereafter

this is a little tricky to integrate into ci

Re: Fuzz testing: the best thing to happen to our application tests

#19

Post by PhilipRoman »

todd8 wrote: post #9

I was working on Distributed Services for AIX in 1986 and 1987, a distributed filesystem to compete with NFS. As this was being developed by a dev team, my colleague and I pondered how to test the system that we had architected. There were so many possible states that a system's file system can be in. Were the conventional tests going to catch subtle bugs? Here's an example of an unusual but not unheard of issue: in…

I used this strategy for implementing a regex engine. I wanted to completely imitate the Lua pattern implementation, so I generated random patterns, ran them on random strings and compared the results.

It was very pleasant to work with such a system. Nowadays I would probably fuzz the patterns with AFL somehow.

Re: Fuzz testing: the best thing to happen to our application tests

#20

Post by farresito »

jacquesm wrote: post #11
todd8 wrote: post #9

I was working on Distributed Services for AIX in 1986 and 1987, a distributed filesystem to compete with NFS. As this was being developed by a dev team, my colleague and I pondered how to test the system that we had architected. There were so many possible states that a system's file system can be in. Were the conventional tests going to catch subtle bugs? Here's an example of an unusual but not unheard of issue: in…

That's very much my experience: asymptotic reduction in bug incidence and matching increase in runtime between subsequent errors. To the point that they are so rare that you think you have fixed all of them. But that's usually an illusion: it's just that the error rate is now so low that you no longer observe incidents yourself. The only way to get past that hurdle is to do the bad thing: release and hope that you go…

> The better way is to design it from the ground up with a bunch of instrumentation that keeps all of your invariants under close observation and that stops the moment anything is not according to your assumptions.

Any personal "war stories" you are willing to share explaining how you went about designing such a system? :-) Or any presentation of someone who did that?

Post reply on HN