Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we applied fuzzing techniques to cURL

blog.trailofbits.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: How we applied fuzzing techniques to cURL

#21

Post by not2b »

martinky24 wrote: post #4
not2b wrote: post #2

[flagged]

The analogy sounds great and catchy (and sufficiently contrarian to get knee jerk upvotes from the site), but it really oversimplifies things and misses the mark on what types of problems fuzzing catches and why that's incredibly important in a project as popular as curl.

I agree that fuzzing is a very important technique, and as I said it's worth fixing any issues found, but I still think that many of the most common uses of curl are very problematic. As you can see, I didn't get any knee jerk upvotes.

Re: How we applied fuzzing techniques to cURL

#22

Post by nickpsecurity »

cpill wrote: post #12

[flagged]

Here’s a survey of fuzzing techniques:

https://arxiv.org/pdf/1812.00140.pdf

One of the earliest examples was a person who fuzzed many applications by getting them to load files he created. His files triggered many vulnerabilities in apps. His method (IIRC): take a file, flip a bit, load it, repeat throughout the file. Amazing how effective it was.

Re: How we applied fuzzing techniques to cURL

#23

Post by [deleted] »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

[deleted]

Re: How we applied fuzzing techniques to cURL

#25

Post by Attummm »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

For libraries, tools, and frameworks, testing is crucial as it ensures that the code relying on them can address the issue at hand. Code can only be as reliable as what it's leaning on. So, to answer your question, a lot of time.

In a business-oriented project(at most jobs), code may undergo frequent changes due to requests from business, thus too much focus on testing could potentially slowing down development speed if extensive testing is implemented for each change. However, regression tests can still provide valuable insights and allow for faster development later in the life of the project.

While many projects only focus on happy path testing, the use of such tests might not be as high. Coupling them with Negative Testing, and even better, implementing boundary testing, compels developers to consider both valid and invalid inputs, helping to identify and address potential edge cases before they become bugs or security issues in production.

For instance, this [0] codebase has more tests than actual code, including fuzzing tests.

[0]https://github.com/Attumm/redis-dict/blob/main/tests.py

Re: How we applied fuzzing techniques to cURL

#26

Post by hgs3 »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Code that isn't tested, isn't done. Tests not only verify the expectations, but also prevent future regression. Fuzzing is essential for code that accepts external inputs. Heartbleed was discoverable with a fuzzer.

Re: How we applied fuzzing techniques to cURL

#27

Post by 1over137 »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

>At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Depends if you are writing software to control a pacemaker, or writing software for some silly smartphone game.

Re: How we applied fuzzing techniques to cURL

#28

Post by pests »

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Everyone always praises SQLite

> As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code [...] the project has 590 times as much test code and test scripts - 92053.1 KSLOC.

https://www.sqlite.org/testing.html

Re: How we applied fuzzing techniques to cURL

#29

Post by guerrilla »

pests wrote: post #28
SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

Everyone always praises SQLite > As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code [...] the project has 590 times as much test code and test scripts - 92053.1 KSLOC. https://www.sqlite.org/testing.html

What the fuck. If they're investing that much then why don't they just go straight to formal verification. This is what things like frama-c (or whatever's popular now) are for.

Re: How we applied fuzzing techniques to cURL

#30

Post by gavinhoward »

guerrilla wrote: post #29
pests wrote: post #28

Earlier quoted context omitted.

Everyone always praises SQLite > As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code [...] the project has 590 times as much test code and test scripts - 92053.1 KSLOC. https://www.sqlite.org/testing.html

What the fuck. If they're investing that much then why don't they just go straight to formal verification. This is what things like frama-c (or whatever's popular now) are for.

The problem is that effort to do formal verification goes exponential beyond a certain point.

seL4 is around 10-12 KLoC, and it took a decade of effort from multiple people to make it happen.

At the size of SQLite, especially where they have to operate on platforms with different behavior (as an OS, seL4 is the platform), formal verification is just too much effort.

All that said, your reaction is totally understandable.

Post reply on HN