Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we applied fuzzing techniques to cURL

blog.trailofbits.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: How we applied fuzzing techniques to cURL

#71

Post by rwmj »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Fuzzing isn't really a "2-3 times longer" thing. You have to set it up once (which for basic fuzzing is quite easy), you throw compute power at it, and it turns up bugs for you. As you make changes to the project the fuzzer will explore new paths mostly automatically.

You may want to spend some time looking at code coverage and doing some of the advanced things outlined in this article, especially for very high risk / reward projects like Curl, but even that is not a lot of work.

Re: How we applied fuzzing techniques to cURL

#72

Post by saagarjha »

DylanSp wrote: post #51
yjftsjthsd-h wrote: post #20

Earlier quoted context omitted.

In the case of curl, the cost/benefit analysis is probably skewed by it being deployed on a massive scale, such that any bugs in curl have an unusually large impact. If your company's in-house CRM has an exploitable bug, that's bad but the impact is just your company. If libcurl has an exploitable bug, that's millions of devices affected.

Probably billions of devices at this point, honestly.

Curl claims 20 billion installations, FWIW.

Re: How we applied fuzzing techniques to cURL

#74

Post by PoignardAzur »

gavinhoward wrote: post #16
PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid…

Your post doesn't address my question (and neither do any of the others replies). The article says:

> If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked.

Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage?

If it's a different valid TLV value each time, then it's not really a dummy value, is it? In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start?

Re: How we applied fuzzing techniques to cURL

#75

Post by aaron695 »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

[dead]

Re: How we applied fuzzing techniques to cURL

#76

Post by aaron695 »

PoignardAzur wrote: post #74
gavinhoward wrote: post #16

Earlier quoted context omitted.

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid…

Your post doesn't address my question (and neither do any of the others replies). The article says: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? If it's a different valid TLV v…

[dead]

Re: How we applied fuzzing techniques to cURL

#77

Post by IshKebab »

SAI_Peregrinus wrote: post #64
tonyarkles wrote: post #40

Earlier quoted context omitted.

Link to how SQLite is tested, for anyone who's curious: https://www.sqlite.org/testing.html There's also an interesting thing where formal verification requires a formal specification, which afaik there isn't one for SQLite. One of the toughest problems that someone would run into trying to put together a formal specification for code as widely deployed as SQLite boils down to Hyrum's Law[1]: on a long enough time sc…

Also a formal specification can have bugs. Formal verification checks that the code matches the spec, not that the spec implements all desired behaviors and no undesired behaviors.

You're right, though it's worth noting that you can also use formal verification to verify properties about the specification! For example you can verify that a security system doesn't allow privilege escalation.

Re: How we applied fuzzing techniques to cURL

#78

Post by williamdclt »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

It’s the 101 of talking about testing, it’s been discussed to the death. As usual, “it’s a trade-off”.

You’re right that writing tests can make shipping a feature slower. However, already having tests makes shipping a feature faster (higher confidence it works, less manual testing of the feature and the adjacent ones). It also lowers the amount of bugs, which are going to slow down delivery of new features. They also increase maintenance cost though (tests are code that needs maintenance too).

Is it worth it? Depends, although it’s quite rare that no test at all is the right trade off

Re: How we applied fuzzing techniques to cURL

#79

Post by costco »

OSS-fuzz is great in many cases but as this post hints at it seems like a lot of the time the fuzzers miss things because no one actually reviews the coverage info. Not in this case, but sometimes it's like someone just found the functions with signatures like parse(unsigned char *data, size_t size) just to get the money from Google for integrating fuzzing.

Then again some things are just difficult to fuzz properly. I tried writing a libpurple IRC harness by doing it the "right way" with the 3000 libpurple event callbacks you are supposed to set up and structures you are supposed to allocate, which worked, but it was very slow. I ended up being lazy and only calling irc_parse_msg after modifying the code to remove error logging that required proper libpurple setup.

Re: How we applied fuzzing techniques to cURL

#80

Post by ackbar03 »

gavinhoward wrote: post #16
PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid…

You think generative ai/deep learning tools could help with this?
Post reply on HN