Bug finding is slow in spite of many eyeballs
daniel.haxx.se
Bug finding is slow in spite of many eyeballs
1–10 of 69 posts
Re: Bug finding is slow in spite of many eyeballs
#2Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.
Re: Bug finding is slow in spite of many eyeballs
#3Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here:
http://www.catb.org/esr/writings/homesteading/cathedral-baza...
Re: Bug finding is slow in spite of many eyeballs
#4Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.
And that is the key IMHO. Closed-source requires enough commercial incentive by one firm to look at it. Open-source requires enough incentive in the aggregate across many problem, each of whom may have their own reasons... and who cares what they are.
Most bugs I have seen figured out have been a collaborative effort. One person finds one part, which leads to the next person figuring something else out, etc. Much harder to do when it is just your job, and you may not even be paid for this work.
Re: Bug finding is slow in spite of many eyeballs
#5Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.
Closed source often has eyeballs specifically paid to... eyeball it.
Re: Bug finding is slow in spite of many eyeballs
#6Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.
Closed source often has eyeballs specifically paid to... eyeball it.
maybe? How can we be sure?
Re: Bug finding is slow in spite of many eyeballs
#7While technically, it is possible to use statistics as "proof" with N=30, but it is stretching it a bit, IMHO.
For example, stating that the amount of reports per year corresponds to the amount code added, by stating that both are "somewhat linear" is not very solid. I could just as well state that the amount of reports per year is "somewhat exponential" and conclude that it does not correspond to the amount of lines of code added.
This does not make the point overall any less true, it is just that the foundation: the numbers, are too few to make any grand conclusions from.
Re: Bug finding is slow in spite of many eyeballs
#8Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...
Yes, that's the party line. If it was true, the number of open bugs would decrease over time. For most open source programs, it increases.
Mozilla has passed the 1 million bug mark.
Re: Bug finding is slow in spite of many eyeballs
#9Whereas closed source proprietary software has no eyeballs on it. Open source is at least an opportunity to identify problems by third parties without reverse-engineering. Open source also allows code analysis tools to do automated tests across wide numbers of codebases.
Closed source often has eyeballs specifically paid to... eyeball it.
All four combinations exist in nature: Open source code with plenty of eyeballs, open source software with no eyeballs, closed source software with tons of reviewers, and closed source software with no no reviewers. It'd be interesting to see what the percentage break down is amongst these, but it probably wouldn't be surprising.
It's also another tick in the box as to why pre-checkin code review is so important - bad code is often immortal, and it can be really hard to patch out bugs if people have grown to rely on broken behavior, so it's best not to get yourself in that state to begin with.
Re: Bug finding is slow in spite of many eyeballs
#10Linus' Law is not some kind of catch all that applies to auditing code for security weaknesses. It specifically refers to the rapid quality control that happens when you release early and release often - the bazaar method of software development, as outlined here: http://www.catb.org/esr/writings/homesteading/cathedral-baza...
Given enough eyeballs, all bugs are shallow.
Given the universal quantifier, one would assume that ESR meant security bugs here as well.
Also, it is stated as a law, but ESR did not back it up with much statistical evidence to support it. I would be surprised if it holds in reality for all bugs.
Extra questions:
- Does releasing early and often lead to more (non-shallow) eyeballs? Not really: see Heartbleed or the OpenSSL RNG bug in Debian.
- Do you need more eyeballs, or just the right eyeballs? (Given the right eyeballs, all bugs are shallow)
- Isn't the effect of moving to a safer language larger?