How we applied fuzzing techniques to cURL • Hacker News Board

Re: How we applied fuzzing techniques to cURL

#81

Post by SAI_Peregrinus » Sat, Mar 02, 2024, 5:25 PM UTC

IshKebab wrote: post #77

SAI_Peregrinus wrote: post #64

Earlier quoted context omitted.

Also a formal specification can have bugs. Formal verification checks that the code matches the spec, not that the spec implements all desired behaviors and no undesired behaviors.

You're right, though it's worth noting that you can also use formal verification to verify properties about the specification! For example you can verify that a security system doesn't allow privilege escalation.

Yep, though of course it's still on you to remember to verify everything you care about. Actually writing out all your requirements in a formal (or just programming) language is the hard part of programming.

Re: How we applied fuzzing techniques to cURL

#82

Post by MajesticHobo2 » Sat, Mar 02, 2024, 8:35 PM UTC

PoignardAzur wrote: post #74

gavinhoward wrote: post #16

Earlier quoted context omitted.

Not an expert, but I am a power user of fuzzers. The problem is that the space of invalid inputs is far larger than the space of valid inputs. Sometimes orders of magnitude larger, say billions or more invalid inputs to one valid input. Naive fuzzing will hit so many error cases that it will hardly produce a valid input. For the ratio that I mentioned, you might run a fuzzer for a billion runs and only get one valid…

Your post doesn't address my question (and neither do any of the others replies). The article says: > If the data can’t be parsed into a valid TLV, instead of throwing it away, return a syntactically correct dummy TLV. This can be anything, as long as it can be successfully unpacked. Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage? If it's a different valid TLV v…

> Is the syntactically correct dummy value the same each time? If so, how does that lead to new coverage?

Per my understanding, the dummy value is constant but is only used once (to create the first valid TLV). Everything after that is a mutation of the original value that, due to the custom mutator logic, is a valid TLV. The mutations are where new coverage comes from.

> In any case, why bother with this "if invalid replace with dummy" step? Why not generate/mutate a valid TLV value from the start?

I'm guessing to handle both when there are seed files that are already valid (which you'll want to use instead of a dummy value), and when there aren't any valid seed files.

Re: How we applied fuzzing techniques to cURL

#83

Post by pjmlp » Sun, Mar 03, 2024, 10:11 AM UTC

SamuelAdams wrote: post #10

I am curious how much effort goes into creating and maintaining unit tests and fuzzing tests. Sometimes it takes longer / more lines of code to write thorough tests than it does to implement the core feature. At that point, is it worth the time invested? Every new feature can take 2-3 times longer to deliver due to adding tests.

The point where quality and security matters, and not providing them comes with returned products, canceled contracts or possible lawsuits.

Re: How we applied fuzzing techniques to cURL

#84

Post by dogcomplex » Tue, Mar 12, 2024, 10:25 PM UTC

vlovich123 wrote: post #68

dogcomplex wrote: post #43

Earlier quoted context omitted.

I would generally double whatever your expectations are for the initial feature development. Tests are essentially a second implementation from a different angle running in parallel, hoping the results match. Every feature change means changing 2 systems now. You save a bit of subsequent time with easier debugging when other features break tests, but that's somewhat eaten up by maintaining a system twice the size. Th…

Tests are only a second implementation if you use test doubles incorrectly. Test doubles should only be used for I/O outside of the program under test that you can’t really run locally / is a network dependency (eg mocking a SaaS service or something) or for performance (mocking database responses vs spinning up a test database instance). If you do it write, most of your tests are just testing each layer and everythi…

I guess I'm referring here to the unit test standards that seem typically employed by automated code review frameworks - i.e. "100% coverage" checkers. With those, any subsequent change to the system requires understanding and modifying both the original code and the set of tests and seems to end up being around double the effort. It's not actual duplication - but mocking expected inputs/outputs takes on its own system (and often programming language in most frameworks) which is not always trivial. You may be referring to something different though.

In those situations, yes I stand by them being a costly overhead - which makes sense in large collaborative systems, but not so much in small agile MVPs.