Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we applied fuzzing techniques to cURL

blog.trailofbits.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: How we applied fuzzing techniques to cURL

#61

Post by pdimitar »

jnwatson wrote: post #60
hgs3 wrote: post #26

Earlier quoted context omitted.

Code that isn't tested, isn't done. Tests not only verify the expectations, but also prevent future regression. Fuzzing is essential for code that accepts external inputs. Heartbleed was discoverable with a fuzzer.

In most situations, we don't get paid to make code "done'. We get paid to get it close enough to work most of the time.

True but cURL is in a very different situation compared to your average run-of-the-mill program. It's one of the most massively deployed open-source software out there. When you are not beholden to capricious shareholders and incompetent managers you can actually focus on quality.

Re: How we applied fuzzing techniques to cURL

#62

Post by evil-olive »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

> Every new feature can take 2-3 times longer to deliver due to adding tests.

alternate framing: you can have a feature done in half or a third of the time, if you don't care whether it actually works or not

...or whether it will continue working, as other new features are added to the system and the codebase evolves and grows. that sort of maintainability is one of the key things you get from investing in writing tests - you can refactor without fear of breaking some existing feature.

it also allows you to hire engineers and give them a "safety net" where they can make changes and be confident they're not breaking some crucial functionality of the unfamiliar codebase.

done correctly (an important caveat, because lots of people do testing poorly and then conclude that all testing is bad) that time spent writing tests is not wasted effort. cutting corners by skipping tests is very much a false economy in the long term.

Re: How we applied fuzzing techniques to cURL

#64

Post by SAI_Peregrinus »

tonyarkles wrote: post #40
gavinhoward wrote: post #30

Earlier quoted context omitted.

The problem is that effort to do formal verification goes exponential beyond a certain point. seL4 is around 10-12 KLoC, and it took a decade of effort from multiple people to make it happen. At the size of SQLite, especially where they have to operate on platforms with different behavior (as an OS, seL4 is the platform), formal verification is just too much effort. All that said, your reaction is totally understanda…

Link to how SQLite is tested, for anyone who's curious: https://www.sqlite.org/testing.html There's also an interesting thing where formal verification requires a formal specification, which afaik there isn't one for SQLite. One of the toughest problems that someone would run into trying to put together a formal specification for code as widely deployed as SQLite boils down to Hyrum's Law[1]: on a long enough time sc…

Also a formal specification can have bugs. Formal verification checks that the code matches the spec, not that the spec implements all desired behaviors and no undesired behaviors.

Re: How we applied fuzzing techniques to cURL

#65

Post by mkj »

PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

You want the failures to occur deeper in the programs so you cover more code paths.

Re: How we applied fuzzing techniques to cURL

#66

Post by pests »

tonyarkles wrote: post #40
gavinhoward wrote: post #30

Earlier quoted context omitted.

The problem is that effort to do formal verification goes exponential beyond a certain point. seL4 is around 10-12 KLoC, and it took a decade of effort from multiple people to make it happen. At the size of SQLite, especially where they have to operate on platforms with different behavior (as an OS, seL4 is the platform), formal verification is just too much effort. All that said, your reaction is totally understanda…

Link to how SQLite is tested, for anyone who's curious: https://www.sqlite.org/testing.html There's also an interesting thing where formal verification requires a formal specification, which afaik there isn't one for SQLite. One of the toughest problems that someone would run into trying to put together a formal specification for code as widely deployed as SQLite boils down to Hyrum's Law[1]: on a long enough time sc…

That's the same link I posted in the original comment, JSYK.

Re: How we applied fuzzing techniques to cURL

#67

Post by vlovich123 »

gavinhoward wrote: post #16
PoignardAzur wrote: post #13

I don't get the part about custom mutators: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. If you're creating a dummy value, how is that better than failing? How does that give your fuzzer better coverage?

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid…

I personally would separate that into two separate fuzz cases - one that generates only valid inputs and one that generates only invalid inputs and spend more resources on verifying the latter because validating invalid inputs is more important. I didn’t read the article but I like property testing for this where your mutator takes random values and uses that to generate a valid input somehow rather than stubbing in a static value. Where I can use a static value is where I wouldn’t be fuzzing the validation of that value. Of course certs are complicated beasts so I’m sure the cURL people did what made sense to them.

Re: How we applied fuzzing techniques to cURL

#68

Post by vlovich123 »

dogcomplex wrote: post #43
SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

I would generally double whatever your expectations are for the initial feature development. Tests are essentially a second implementation from a different angle running in parallel, hoping the results match. Every feature change means changing 2 systems now. You save a bit of subsequent time with easier debugging when other features break tests, but that's somewhat eaten up by maintaining a system twice the size. Th…

Tests are only a second implementation if you use test doubles incorrectly. Test doubles should only be used for I/O outside of the program under test that you can’t really run locally / is a network dependency (eg mocking a SaaS service or something) or for performance (mocking database responses vs spinning up a test database instance). If you do it write, most of your tests are just testing each layer and everything below it.

I have yet to see a case where omitting tests actually helps you move meaningfully faster - you’re probably generating more heat than light and that makes you feel like you’re moving faster.

Re: How we applied fuzzing techniques to cURL

#70

Post by kgeist »

hgs3 wrote: post #26
SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Code that isn't tested, isn't done. Tests not only verify the expectations, but also prevent future regression. Fuzzing is essential for code that accepts external inputs. Heartbleed was discoverable with a fuzzer.

>Code that isn't tested, isn't done. Tests not only verify the expectations, but also prevent future regression

A few weeks ago someone at the company implemented a feature without tests. It worked perfectly, everyone was happy. A release of another new feature by a different team a few days later broke the previous feature :)

Post reply on HN