Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we applied fuzzing techniques to cURL

blog.trailofbits.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: How we applied fuzzing techniques to cURL

#11

Post by epistasis »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

There's a tradeoff for sure, but to fully evaluate that tradeoff you must also take into account the future time spent on the code base, including the amount of time needed to debug future features, adopt a new build environment, allow new team members to muck around and see what does and does not work...

If it's a small project that won't be extended much, then perhaps then re-runnable unit tests may not make the bar as a net positive tradeoff. But some time must still be spent testing the code, even if it's not tests that are written in code.

Re: How we applied fuzzing techniques to cURL

#13

Post by PoignardAzur »

I don't get the part about custom mutators:

> If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked.

If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

Re: How we applied fuzzing techniques to cURL

#14

Post by avgcorrection »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

The best-case IMO is a test suite where writing the test is almost as easy (or maybe easier?) than doing the equivalent manual test. The test code is maybe 2–4x longer than the change.

The worst case is… arbitrarily bad, to the point of being impossible. Test setup is hell because there are too many dependencies. You have to mock the “real world” since (again) things depend on other things too much and you can’t really do a simple string-manipulating change without setting up a whole environment. Also you introduce bugs in your tests that take longer to debug than the real change you are making.

What I feel that we (current project) have fallen into the trap of is that there wasn’t a test suite from the start. Just manual testing. Then when you get to a stage where you feel you need one, the code base is not structured to make automated testing simple.

Re: How we applied fuzzing techniques to cURL

#15

Post by pstrateman »

PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

The file format they choose is difficult for a fuzzer to produce valid examples by random chance.

The file format isn't what's being fuzzed, so trying to accept as many things as possible as valid is useful.

It's a trick to make the fuzzer faster.

Re: How we applied fuzzing techniques to cURL

#16

Post by gavinhoward »

PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

Not an expert, but I am a power user of fuzzers.

The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input.

Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid input in the bunch.

Using a custom mutator and returning a dummy value will give the fuzzer a starting point from a valid input and makes generating other valid inputs more likely.

For my part, I prefer to use custom mutators to generate valid test cases most of the time, but I want some invalid inputs because error handling is where most bugs are.

Re: How we applied fuzzing techniques to cURL

#18

Post by bmcniel »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Don't test protect against future unintended regressions?

If all code was write-only then testing is probably a waste of time but code changes constantly.

Re: How we applied fuzzing techniques to cURL

#19

Post by Jtsummers »

cpill wrote: post #12

[flagged]

Throw random inputs at a system and see what happens. The primary purpose is to trigger crashes or going into some other error state or the like. It's useful for checking the stability/robustness of a system.

https://en.wikipedia.org/wiki/Fuzzing

Re: How we applied fuzzing techniques to cURL

#20

Post by yjftsjthsd-h »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

In the case of curl, the cost/benefit analysis is probably skewed by it being deployed on a massive scale, such that any bugs in curl have an unusually large impact. If your company's in-house CRM has an exploitable bug, that's bad but the impact is just your company. If libcurl has an exploitable bug, that's millions of devices affected.
Post reply on HN