[deleted]
How we applied fuzzing techniques to cURL
41–50 of 84 posts
Re: How we applied fuzzing techniques to cURL
#42[deleted]
Re: How we applied fuzzing techniques to cURL
#43I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.
I would generally double whatever your expectations are for the initial feature development. Tests are essentially a second implementation from a different angle running in parallel, hoping the results match. Every feature change means changing 2 systems now. You save a bit of subsequent time with easier debugging when other features break tests, but that's somewhat eaten up by maintaining a system twice the size.
There are reasons many MVP developers and small teams whose focus is more on rapid feature implementation than large team coordination or code stability forego writing tests. It doesn't make sense in all circumstances. Generally, more complex, less grokable, more large-team-oriented or public library code is when you need testing.
Re: How we applied fuzzing techniques to cURL
#44[deleted]
Re: How we applied fuzzing techniques to cURL
#45Earlier quoted context omitted.
Everyone always praises SQLite > As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code [...] the project has 590 times as much test code and test scripts - 92053.1 KSLOC. https://www.sqlite.org/testing.html
What the fuck. If they're investing that much then why don't they just go straight to formal verification. This is what things like frama-c (or whatever's popular now) are for.
Because formal verification doesn't scale at the moment. There are a relatively few number of experts on formal method around the globe and some of them need to work on SQLite indefinitely unless you're okay with just one-off verification. Frama-C has a different background because it's from Inria, the institution with many formal method experts.
Re: How we applied fuzzing techniques to cURL
#46I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.
It's an economical decision. If you're developing a new indie game with $10k revenue expectation in its lifetime, then probably it's not worth your time. But if it's a core infrastructure of multi-billion dollar business, then yes it's worth your time since any non-trivial security incidents may cost more than your annual salary.
Re: How we applied fuzzing techniques to cURL
#47Earlier quoted context omitted.
Everyone always praises SQLite > As of version 3.42.0 (2023-05-16), the SQLite library consists of approximately 155.8 KSLOC of C code [...] the project has 590 times as much test code and test scripts - 92053.1 KSLOC. https://www.sqlite.org/testing.html
What the fuck. If they're investing that much then why don't they just go straight to formal verification. This is what things like frama-c (or whatever's popular now) are for.
You can do many things other than testing:
- use a memory safe language
- formal verification (multiple implementations even)
- build a simulator like FoundationDB did
Re: How we applied fuzzing techniques to cURL
#48I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.
It depends on how risk tolerant you are. If you are at a startup, the growth of your startup is often largely dependent on how quickly you can add features / onboard customers. In that context, writing tests not only slows you down from adding new features, it might make it harder to modify existing features. In addition, early customers also tend to be accepting of small bugs -- they themselves already have "taken a risk" in trusting an early startup. So testing is not really valuable -- you want to remain agile, and you won't lose much money because of a bug.
On the other hand, if you are Google, you already have found a money-printing firehose, and you /don't/ want to take on any additional unnecessary risk. Any new code needs to /not/ break existing functionality -- if it does, you might lose out of millions of revenue. In addition, your product becomes so large that it is impossible to manually test every feature. In this case, tests actually help you move /faster/ because they help you at scale automatically ensure that a change does not break anything.
While cURL does not make any money, it is solidly on the mature/Google end of the testing spectrum. It has found a footing in the open source tooling "market" and people rely on it to maintain its existing functionality. In addition, it has accumulated a fairly large surface area, so manual testing is not really feasible for every feature. So testing similarly helps cURL developers move faster (in the long run), not slower.
Re: How we applied fuzzing techniques to cURL
#49Earlier quoted context omitted.
The problem is that effort to do formal verification goes exponential beyond a certain point. seL4 is around 10-12 KLoC, and it took a decade of effort from multiple people to make it happen. At the size of SQLite, especially where they have to operate on platforms with different behavior (as an OS, seL4 is the platform), formal verification is just too much effort. All that said, your reaction is totally understanda…
Link to how SQLite is tested, for anyone who's curious: https://www.sqlite.org/testing.html There's also an interesting thing where formal verification requires a formal specification, which afaik there isn't one for SQLite. One of the toughest problems that someone would run into trying to put together a formal specification for code as widely deployed as SQLite boils down to Hyrum's Law[1]: on a long enough time sc…
https://corecursive.com/066-sqlite-with-richard-hipp/ is an interesting interview with the lead developer for SQLite about the inspiration and work that went into that testing as well.
Re: How we applied fuzzing techniques to cURL
#50[flagged]