Bug finding is slow in spite of many eyeballs • Hacker News Board

Re: Bug finding is slow in spite of many eyeballs

#11

Post by reubenmorais » Mon, Feb 23, 2015, 8:48 AM UTC

Animats wrote: post #8

camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

Yes, that's the party line. If it was true, the number of open bugs would decrease over time. For most open source programs, it increases. Mozilla has passed the 1 million bug mark.

> For most open source programs, it increases.

I know of at least one study that contradicts this claim:

"We found that with shorter release cycles, users do not experience significantly more post-release bugs and bugs are fixed faster, yet users experience these bugs earlier during software execution (the program crashes earlier)."

http://swat.polymtl.ca/~foutsekh/docs/Khomh-MSR-2012.pdf

Re: Bug finding is slow in spite of many eyeballs

#12

Post by br3w5 » Mon, Feb 23, 2015, 8:50 AM UTC

ryanlol wrote: post #5

iwwr wrote: post #2

Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.

Closed source often has eyeballs specifically paid to... eyeball it.

But just because they're paid to do it doesn't mean they care, or that they're any good at it.

Re: Bug finding is slow in spite of many eyeballs

#13

Post by rndgermandude » Mon, Feb 23, 2015, 8:50 AM UTC

Animats wrote: post #8

camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

Yes, that's the party line. If it was true, the number of open bugs would decrease over time. For most open source programs, it increases. Mozilla has passed the 1 million bug mark.

To be fair, mozilla uses bugzilla to track everything, not just "bug" bugs. Each new feature in development has a bug, or multiple bugs. When code is refactored there is a bug for that, when somebody wants commit access there is a bug for that, when an employee needs a new laptop there is a bug for that, when a community organizer wants some money or gear for an event there is a bug for that, and so on... Other organizations use their bug/issue trackers in a similar manner.

Re: Bug finding is slow in spite of many eyeballs

#14

Post by wiml » Mon, Feb 23, 2015, 9:11 AM UTC

camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

I always understood this saying to be referring to the difficulty of fixing bugs once known. Some bugs are really hard to understand, reason about, and figure out how to fix. Given a large number of contributors, someone is likely to have just the right mindset and familiarity that the bug is easy for them to understand and fix (it is "shallow" for them).

Re: Bug finding is slow in spite of many eyeballs

#15

Post by amelius » Mon, Feb 23, 2015, 9:23 AM UTC

awalton wrote: post #9

ryanlol wrote: post #5

Earlier quoted context omitted.

Closed source often has eyeballs specifically paid to... eyeball it.

All four combinations exist in nature: Open source code with plenty of eyeballs, open source software with no eyeballs, closed source software with tons of reviewers, and closed source software with no no reviewers. It'd be interesting to see what the percentage break down is amongst these, but it probably wouldn't be surprising. It's also another tick in the box as to why pre-checkin code review is so important - ba…

Also interesting would be a breakdown by language. For example C++ versus Haskell.

Re: Bug finding is slow in spite of many eyeballs

#16

Post by Taek » Mon, Feb 23, 2015, 9:38 AM UTC

microtonal wrote: post #10

camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

Given enough eyeballs, all bugs are shallow. Given the universal quantifier, one would assume that ESR meant security bugs here as well. Also, it is stated as a law, but ESR did not back it up with much statistical evidence to support it. I would be surprised if it holds in reality for all bugs. Extra questions: - Does releasing early and often lead to more (non-shallow) eyeballs? Not really: see Heartbleed or the Op…

This law mostly applies to usability bugs. If a driver isn't working, somebody is going to notice. Security bugs on the other hand are often invisible during all forms of standard use. Heartbleed for example will never disrupt the user experience. The only way to find that problem is to be actively looking for it.

Generally speaking, 'eyeballs' aren't actively looking for problems. They're passively using software and noticing when something seems out of place. Many security bugs will never seem out of place, and are only discoverable when someone is intentionally probing for security bugs.

Re: Bug finding is slow in spite of many eyeballs

#17

Post by JoachimS » Mon, Feb 23, 2015, 9:48 AM UTC

ryanlol wrote: post #5

iwwr wrote: post #2

Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.

Closed source often has eyeballs specifically paid to... eyeball it.

The question is what do the brain behind the eyeballs do?

Do they in fact register the bug, ignore it, file a report, solve it? Without source code we are left to trust the vendor that they actually fins and solve bugs. The vendor and/or its staff might be under pressure to leave bugs in place. We just don't know.

Re: Bug finding is slow in spite of many eyeballs

#18

Post by JoachimS » Mon, Feb 23, 2015, 9:52 AM UTC

br3w5 wrote: post #12

ryanlol wrote: post #5

Earlier quoted context omitted.

Closed source often has eyeballs specifically paid to... eyeball it.

But just because they're paid to do it doesn't mean they care, or that they're any good at it.

Headcount.

Given enough headcount all problems are solved in line with the time plan is the enterprise version of the law. With enough engineers, you can just treat them as replaceable units, just numbers in an Excel sheet. Move them around as needed to meet staffing requirements, or resource allocation requests as they are often called.

(No, I'm not cynic at all.)

Re: Bug finding is slow in spite of many eyeballs

#19

Post by icebraining » Mon, Feb 23, 2015, 9:52 AM UTC

microtonal wrote: post #10

camperman wrote: post #3

Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...

Given enough eyeballs, all bugs are shallow. Given the universal quantifier, one would assume that ESR meant security bugs here as well. Also, it is stated as a law, but ESR did not back it up with much statistical evidence to support it. I would be surprised if it holds in reality for all bugs. Extra questions: - Does releasing early and often lead to more (non-shallow) eyeballs? Not really: see Heartbleed or the Op…

- Does releasing early and often lead to more (non-shallow) eyeballs? Not really: see Heartbleed or the OpenSSL RNG bug in Debian.

The existence of those bugs doesn't disprove the claim. You'd need to show that software that is not released early and often doesn't have (even) fewer eyeballs.

Re: Bug finding is slow in spite of many eyeballs

#20

Post by krig » Mon, Feb 23, 2015, 9:59 AM UTC

"Because in reality, many many bugs are never really found by all those given “eyeballs” in the first place. They are found when someone trips over a problem and is annoyed enough to go searching for the culprit, the reason for the malfunction."

That /someone/ is able to go searching for the culprit instead of having to rely on someone ELSE to look at the source and figure out what is going on is the whole point of the quote, no?