Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug finding is slow in spite of many eyeballs

daniel.haxx.se

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Bug finding is slow in spite of many eyeballs

#41

Post by gordaco »

scott_s wrote: post #40
krig wrote: post #20

"Because in reality, many many bugs are never really found by all those given “eyeballs” in the first place. They are found when someone trips over a problem and is annoyed enough to go searching for the culprit, the reason for the malfunction." That /someone/ is able to go searching for the culprit instead of having to rely on someone ELSE to look at the source and figure out what is going on is the whole point of t…

Yes, the "many eyeballs" is a part of it. But I think the author's point is that they are only half. The other half is someone has to trip over the bug. That is, people aren't finding these bugs purely through reading the source code.

There is a huge difference between a code review and the specific search for the cause of a known, defined bug. Should we rephrase Linus' law as "given enough eyeballs, all bugs' causes are shallow"? But maybe that's what the sentence actually meant for the first person that wrote it.

Re: Bug finding is slow in spite of many eyeballs

#42

Post by jpollock »

I've got some different take-a-ways. First, time-to-discovery is different from the shallowness of the bug. Increasing the number of eyeballs looking at the code increases the likelihood that:

  a) someone will encounter the problem.
  b) that someone will be interested enough to dig into it.
  c) they will then find the problem before giving up.
That's the power of many eyeballs. It's an expression of interest. With fewer eyeballs, you might get bug reports (a), but not have enough people looking to get (b+c) out of the community. With closed source, (b+c) can _only_ be provided by the team. Open Source means that this can be provided by a sufficiently large community.

That open source projects are having security bugs reported can probably be explained by economics. It's becoming harder to find security problems in closed source projects (windows), so researchers shift to open source ones. With an open source project, there's a lot of low hanging fruit to be had with static analysis, copied code and fuzzing. Closed source is a lot harder, so people go for where the ROI is good - either towards the rewards or towards open source for practice.

Finally, shifting to Java just shifts the attack to the JVM, which is just as hard to secure. I still remember the year of Java exploits, complete with a remote DoS attack based around sending a floating point number to the server[1]. There will always be bugs. If your goal is to write secure software, open source is good. If your goal is to avoid bad press, making it expensive to test is probably the way to go instead.

[1] http://www.oracle.com/technetwork/topics/security/alert-cve-...

Re: Bug finding is slow in spite of many eyeballs

#43

Post by fdej »

Eventually, we need to move towards formally verified software. Some bugs slip past any number of human eyeballs. It won't be feasible to do formal verification of all software anytime soon, but it should be done for operating system kernels, networking and crypto libraries, virtual machines, compilers, and similar security-critical software.

Switching to programming languages with type systems powerful enough to protect against common classes of bugs is a good first step (but not enough).

Re: Bug finding is slow in spite of many eyeballs

#44

Post by JoachimS »

ukigumo wrote: post #23
JoachimS wrote: post #17

Earlier quoted context omitted.

The question is what do the brain behind the eyeballs do? Do they in fact register the bug, ignore it, file a report, solve it? Without source code we are left to trust the vendor that they actually fins and solve bugs. The vendor and/or its staff might be under pressure to leave bugs in place. We just don't know.

You have to trust the vendor's interest in making a profit. Nothing wrong with that :-) Closed source (usually) gives you Ts&Cs, support model and some level of guarantee of fitness for purpose which is why we pay to use it. This is not to say that there aren't bugs that seem to take forever to fix, but usually that is not the case at least not when there is a risk of the product becoming unsalable. Of course there a…

In these days wit SuperFishes and Lenovos selling out their customers for a few bucks more, trusting that the vendor have an interest in making a profit might not by itself be ok.

Re: Bug finding is slow in spite of many eyeballs

#45

Post by JoachimS »

jpollock wrote: post #42

I've got some different take-a-ways. First, time-to-discovery is different from the shallowness of the bug. Increasing the number of eyeballs looking at the code increases the likelihood that: a) someone will encounter the problem. b) that someone will be interested enough to dig into it. c) they will then find the problem before giving up. That's the power of many eyeballs. It's an expression of interest. With fewer…

And: c) the bug report and fix will not be ignored.

Considering how OpenSSL handled bug reports and fixes (by letting them sit in the tracker for years) or Ulrich Dreppers lets say less than welcoming attitude the works done by the eyes can be a waste. And the many eyes soon withers to nil.

Re: Bug finding is slow in spite of many eyeballs

#46

Post by JoachimS »

chris_wot wrote: post #37
JoachimS wrote: post #21

I've always read the expression backwards: With enough eyeballs, the shallow bugs will be found. Things like spelling errors in print statements, comments etc gets corrected. By off by one errors, use after free and all kinds of subtle logical problems that only manifests once in a while and after long execution will require a focused effort to find (and more and more, good tools.) Daniels description matches this se…

At least you've never read it as "With enough bugs, the shallow eyeballs will be found".

True. Nor that enough eyeballs with bugs will be found at the shallow end.

Re: Bug finding is slow in spite of many eyeballs

#47

Post by bzbarsky »

Animats wrote: post #8
camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

Yes, that's the party line. If it was true, the number of open bugs would decrease over time. For most open source programs, it increases. Mozilla has passed the 1 million bug mark.

Mozilla has passed the 1 million bugs _ever filed_ mark. Across all Mozilla products, for that matter.

Some possibly more useful statistics:

For the "Core" product (read: Gecko), there are as of right now 52k open bugs and 230k closed bugs.

For the "Firefox" product (desktop, not Firefox android), there are 21k open bugs and 128k closed bugs.

Note that these "bugs" include feature requests, so unless you think the number of requested features will decline over time....

Re: Bug finding is slow in spite of many eyeballs

#48

Post by mcdoug »

fdej wrote: post #43

Eventually, we need to move towards formally verified software. Some bugs slip past any number of human eyeballs. It won't be feasible to do formal verification of all software anytime soon, but it should be done for operating system kernels, networking and crypto libraries, virtual machines, compilers, and similar security-critical software. Switching to programming languages with type systems powerful enough to pro…

I feel like if we did this we'd never get to an actual working system. How do you formally verify that a remote file system interacting with a faulty hard disk is doing the right thing? This reminds me of Hurd, which is arguably a superior design that will never be finished.

Re: Bug finding is slow in spite of many eyeballs

#49

Post by fdej »

mcdoug wrote: post #48
fdej wrote: post #43

Eventually, we need to move towards formally verified software. Some bugs slip past any number of human eyeballs. It won't be feasible to do formal verification of all software anytime soon, but it should be done for operating system kernels, networking and crypto libraries, virtual machines, compilers, and similar security-critical software. Switching to programming languages with type systems powerful enough to pro…

I feel like if we did this we'd never get to an actual working system. How do you formally verify that a remote file system interacting with a faulty hard disk is doing the right thing? This reminds me of Hurd, which is arguably a superior design that will never be finished.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic.

Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the functional parts of a crypto library (and demand this standard of future replacements) independently of what goes on in the world of file system drivers.

Realistically speaking, as worse is better, I think a large push for formally verified software probably won't happen for decades to come, but it will happen eventually. It already happened in parts of the hardware industry (Intel certainly don't want the division bug to happen ever again). The main issue is that we need tools that make development easier, to allow keeping up with the rapidly changing requirements in the software world.

Re: Bug finding is slow in spite of many eyeballs

#50

Post by bch »

shin_lao wrote: post #28
netcan wrote: post #24

Despite being big and different and having developed some grumpy old forum symptoms, HN is still smart. As such, I think we have a great example of memetic churn and progression here. We start with manifesto-ey essays trying to cut through some habitual thinking and paradigms. The new fundraising model. Bootstrapping. Web based SAAS is an ocean with a lot of paradigm shifts, the tech options, the business models, the…

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just god damn hard or need to be looked at with the problem in mind. Software quality is orthogonal to the openness or closeness of the source code. I think it is more about the will to do something great, the ability to listen to the user base and of course the te…

> or need to be looked at with the problem in mind.

The article addresses how hard some of the bugs were (they were not hard), but additionally, sometimes (often?) code needs to be looked at without the problem in mind -- who hasn't written code (or a paper, or comment on HN) and proofed it immediately, committed it, and realized it has an error that your mind elided over because you were so sure "what you meant" was there that you glossed over a dumb error. Having a different mind review these things avoids that problem.

Edit: I glossed over this -- I was also referring to proofing your work much later, so it's not still fresh in your mind "as you think it is". So, bring a "different mind" to the table.

s/completely different mind/different mind/

Post reply on HN