Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug finding is slow in spite of many eyeballs

daniel.haxx.se

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Bug finding is slow in spite of many eyeballs

#51

Post by cheatsheet »

netcan wrote: post #24

Despite being big and different and having developed some grumpy old forum symptoms, HN is still smart. As such, I think we have a great example of memetic churn and progression here. We start with manifesto-ey essays trying to cut through some habitual thinking and paradigms. The new fundraising model. Bootstrapping. Web based SAAS is an ocean with a lot of paradigm shifts, the tech options, the business models, the…

It is a long tail distribution and it happens because of different computational complexity classes. I don't really know if there is anything to be done about it.

I'll give you a high grumpy old forum five though.

Re: Bug finding is slow in spite of many eyeballs

#52

Post by wglb »

iwwr wrote: post #2

Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.

Closed source does in fact have eyeballs on it. Vulnerabilities are found in source such as Windows by those who are only looking at the binary.

Automated testing tools are also available at the binary level.

Re: Bug finding is slow in spite of many eyeballs

#53

Post by schoen »

wiml wrote: post #14
camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

I always understood this saying to be referring to the difficulty of fixing bugs once known. Some bugs are really hard to understand, reason about, and figure out how to fix. Given a large number of contributors, someone is likely to have just the right mindset and familiarity that the bug is easy for them to understand and fix (it is "shallow" for them).

This article and your observation remind me that there's a huge gap between ways in which bugs might be "shallow".

If Linus's Law and so on really did rely on people encountering the bugs by chance in everyday use of the software, it's no wonder (in hindsight) this doesn't help with a lot of the security bugs we face, many of which would never be triggered randomly in normal use, but rather require constructing elaborate attack scenarios. Even those that might occur by chance are not likely to be repeatable, and so not likely to get reported or analyzed.

Maybe this points to a change in our prototypical concept of a "bug". When ESR first wrote "The Cathedral and the Bazaar", I would have associated "bug" with something like "the TIFFs produced by program A can't be read by program B", or "program C seems to crash if you have a non-ASCII filename", or "program D drops its network connections if more than 65536 packets are received". Today, I would associate it with something like "an attacker who sends a certificate containing an extension with invalid ASN.1 encoding that follows one or more syntactically valid extensions that are marked Critical and that are unknown to the user-agent can get remote code execution" or "an attacker who sends an XML payload that is parsed correctly by library A and incorrectly by library B due to discrepant handling of Unicode escaping can request operations that should be forbidden".

Well, a literal hot-off-the-presses example would be a bug in handling multibyte characters in the regular expression library in Flash Player, which was exploitable:

http://googleprojectzero.blogspot.com/2015/02/exploitingscve...

In other words, these bugs are often complicated artifacts that require research to find and malice to use -- not annoyances and breakages that are frustrating end users every day.

Re: Bug finding is slow in spite of many eyeballs

#54

Post by wglb »

deitcher wrote: post #4
iwwr wrote: post #2

Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.

And that is the key IMHO. Closed-source requires enough commercial incentive by one firm to look at it. Open-source requires enough incentive in the aggregate across many problem, each of whom may have their own reasons... and who cares what they are. Most bugs I have seen figured out have been a collaborative effort. One person finds one part, which leads to the next person figuring something else out, etc. Much har…

Closed-source requires enough commercial incentive by one firm to look at it.

Or a motivated attacker to reverse it for exploits.

Re: Bug finding is slow in spite of many eyeballs

#55

Post by codehero »

Linus' law may scale, but even assuming these eyeballs produce bug fixes, applying those fixes to the source tree does not scale as well.

I have taken time to put my eyeballs on bugs in spidev's ioctl() and TI's spi driver but my bug fixes are not in the tree.

Signing off, adhering to the source standard and attaining enough respect from the established devs to get your fix accepted are the limiting factors.

I already invested significant amounts of time finding these bugs and fixing these issues; I don't have any more to spend to make Mark Brown or other kernel devs happy.

I don't even care about getting the credit for my fixes, but it seems the kernel devs don't want to take my code to next step and get it integrated.

Re: Bug finding is slow in spite of many eyeballs

#56

Post by cheatsheet »

fdej wrote: post #43

Eventually, we need to move towards formally verified software. Some bugs slip past any number of human eyeballs. It won't be feasible to do formal verification of all software anytime soon, but it should be done for operating system kernels, networking and crypto libraries, virtual machines, compilers, and similar security-critical software. Switching to programming languages with type systems powerful enough to pro…

Sure, but there will still be bugs. We might have to figure out what a bug is semantically on the next layer of abstraction as we are developing it (and have the capacity to formally describe it), but there will be opinions, and thus there will be bugs.

Re: Bug finding is slow in spite of many eyeballs

#57

Post by jdc »

fdej wrote: post #49
mcdoug wrote: post #48

Earlier quoted context omitted.

I feel like if we did this we'd never get to an actual working system. How do you formally verify that a remote file system interacting with a faulty hard disk is doing the right thing? This reminds me of Hurd, which is arguably a superior design that will never be finished.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic. Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the fu…

What are some examples of non-trivial working formally verified software?

Re: Bug finding is slow in spite of many eyeballs

#58

Post by toddkaufmann »

So how many bugs remain?

Mostly rhetorical question, but can any extrapolation be done? If you go back five years, can any of those numbers correlate to the findings since? Do any metrics such as cyclomatic complexity, #defects/kLoC[1][2], unit tests or code coverage help?

In most cases the definition of "defect" is not well-defined, nor in many cases easily comparable (e.g., a typo in a debug message compared to handling SSL flags wrong). Is is a requirements or documentation bug: the specification to the the implementer was not sufficiently clear or was ambiguous. Also, when do we start counting defects? If I misspelled a keyword and the compiler flagged it, does that count? Only after the code is commited? Caught by QA? Or after it is deployed or released in a product?

Is it related to the programming language? Programmer skill level and fluency with language/libraries/tools? Did they not get enough sleep the night before when they coded that section? Or were they deep in thought thinking about 4 edges cases for this method when someone popped their head in to ask about lunch plans and knocked one of them out? Does faster coding == more "productive" programmer == more defects long term?

I'm not sure if we're still programming cavemen or have created paleolithic programming tools yet[3][4].

p.s.: satisified user of cURL since at least 1998!

    [1] http://www.infoq.com/news/2012/03/Defects-Open-Source-Commercial
    [2] http://programmers.stackexchange.com/questions/185660/is-the-average-number-of-bugs-per-loc-the-same-for-different-programming-languag
    [3] https://vimeo.com/9270320 - Greg Wilson - What We Actually Know About Software Development, and Why We Believe It's True
    (probably shorter, more recent talks exists (links appreciated))
    [4] https://www.youtube.com/watch?v=ubaX1Smg6pY - Alan Kay - Is it really "Complex"? Or did we just make it "Complicated"?
    (tangentially about software engineering, but eye-opening for how much more they were doing, and with fewer lines of code) (also, any of his talks)

Re: Bug finding is slow in spite of many eyeballs

#59

Post by fdej »

jdc wrote: post #57
fdej wrote: post #49

Earlier quoted context omitted.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic. Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the fu…

What are some examples of non-trivial working formally verified software?

http://compcert.inria.fr/ is the first that comes to mind.

Re: Bug finding is slow in spite of many eyeballs

#60

Post by fdej »

cheatsheet wrote: post #56
fdej wrote: post #43

Eventually, we need to move towards formally verified software. Some bugs slip past any number of human eyeballs. It won't be feasible to do formal verification of all software anytime soon, but it should be done for operating system kernels, networking and crypto libraries, virtual machines, compilers, and similar security-critical software. Switching to programming languages with type systems powerful enough to pro…

Sure, but there will still be bugs. We might have to figure out what a bug is semantically on the next layer of abstraction as we are developing it (and have the capacity to formally describe it), but there will be opinions, and thus there will be bugs.

Of course. In the end, we don't even have a perfect understanding of the physics on which digital technology is built. Nonetheless, we can evidently do much better than we do today, and delegate more of the hard work to our robotic slaves.
Post reply on HN