Live data from Hacker News

Bug finding is slow in spite of many eyeballs

daniel.haxx.se

21–30 of 69 posts

Re: Bug finding is slow in spite of many eyeballs

#21
I've always read the expression backwards: With enough eyeballs, the shallow bugs will be found.

Things like spelling errors in print statements, comments etc gets corrected. By off by one errors, use after free and all kinds of subtle logical problems that only manifests once in a while and after long execution will require a focused effort to find (and more and more, good tools.) Daniels description matches this sentiment.

Re: Bug finding is slow in spite of many eyeballs

#22
I've been using open source for ages, and have rarely taken the time to look at the source.

But, at least its there.

I sometimes wish there wasn't such motivation to hack software. If we were all working towards a common good (in my case I'd like to see us doing a bit of space exploration, renewable energy tech etc), and we wouldn't need to exploit stuff.

But, we're humans. Greedy little scumbags hehe, always looking for the short term gain. All varying forms of politicians.

Re: Bug finding is slow in spite of many eyeballs

#23
post #5

Earlier quoted context omitted.

Closed source often has eyeballs specifically paid to... eyeball it.

The question is what do the brain behind the eyeballs do? Do they in fact register the bug, ignore it, file a report, solve it? Without source code we are left to trust the vendor that they actually fins and solve bugs. The vendor and/or its staff might be under pressure to leave bugs in place. We just don't know.

You have to trust the vendor's interest in making a profit. Nothing wrong with that :-)

Closed source (usually) gives you Ts&Cs, support model and some level of guarantee of fitness for purpose which is why we pay to use it. This is not to say that there aren't bugs that seem to take forever to fix, but usually that is not the case at least not when there is a risk of the product becoming unsalable.

Of course there are closed source software or appliances that just are not worth their cost because they fail to mitigate the risks of having us trust what we cannot see, but these usually fail to gain any significant market share.

Re: Bug finding is slow in spite of many eyeballs

#24
Despite being big and different and having developed some grumpy old forum symptoms, HN is still smart. As such, I think we have a great example of memetic churn and progression here. We start with manifesto-ey essays trying to cut through some habitual thinking and paradigms. The new fundraising model. Bootstrapping. Web based SAAS is an ocean with a lot of paradigm shifts, the tech options, the business models, the development cycle. Those get people excited and defensive and there's fun to be had arguing it out. There's all sorts of pedantic arguing in the fringes pointing out that this things not strictly true. Then there are manifesto busting pieces. Case studies and anecdotes about manifesto-ed approaches going terribly wrong.

I'm not down on this process, it's progress.

Anyway, I think this is a part of it. "Enough eyeballs" is a big hairy hefty concept, part of the big hairy open sourced concept. No one could have predicted open source. The cultural and economic (BTW, economy is a big part part of culture) machinations of open source add up to something big and impressive and unpredictable. "Enough eyeballs" is a (somewhat squishy) big concept trying to make sense of it. This is like trying to boil down 'platform wars' or 'platform openness' into slogans and roes of thumb. You may come up with an interesting, informative and useful way of looking at things. But, underneath it is a big harry tangle of machinations, circumstances and ungodly complexity. Hence the need for rules of thumb and manifestos in the first place.

This post is part of the more mature stage. The audience is expected to be on Linus' side already and the novel part is discovering that the rule of thumb is isn't working in all cases. Diging into the machinations. Who are all these eyeballs? What are these bugs? Are some types of eyeballs more or less effective against some bugs?

I'm not sure where I'm going with this… be excellent to each other?

Re: Bug finding is slow in spite of many eyeballs

#25
post #23

Earlier quoted context omitted.

The question is what do the brain behind the eyeballs do? Do they in fact register the bug, ignore it, file a report, solve it? Without source code we are left to trust the vendor that they actually fins and solve bugs. The vendor and/or its staff might be under pressure to leave bugs in place. We just don't know.

You have to trust the vendor's interest in making a profit. Nothing wrong with that :-) Closed source (usually) gives you Ts&Cs, support model and some level of guarantee of fitness for purpose which is why we pay to use it. This is not to say that there aren't bugs that seem to take forever to fix, but usually that is not the case at least not when there is a risk of the product becoming unsalable. Of course there a…

>You have to trust the vendor's interest in making a profit. Nothing wrong with that :-)

Sometimes there are conflicts of interest. We have documented cases of 3-letter agencies paying companies to leave bugs or unsafe options in code. Sometimes the backdoors may be more valuable than the product itself.

I suspect many corporate clients found out these days that their SSL MITM software they used made their infrastructure vulnerable.

Re: Bug finding is slow in spite of many eyeballs

#26
post #25
post #23

Earlier quoted context omitted.

You have to trust the vendor's interest in making a profit. Nothing wrong with that :-) Closed source (usually) gives you Ts&Cs, support model and some level of guarantee of fitness for purpose which is why we pay to use it. This is not to say that there aren't bugs that seem to take forever to fix, but usually that is not the case at least not when there is a risk of the product becoming unsalable. Of course there a…

>You have to trust the vendor's interest in making a profit. Nothing wrong with that :-) Sometimes there are conflicts of interest. We have documented cases of 3-letter agencies paying companies to leave bugs or unsafe options in code. Sometimes the backdoors may be more valuable than the product itself. I suspect many corporate clients found out these days that their SSL MITM software they used made their infrastruc…

This is very true. However trusting that open source does not include such vulnerabilities is a leap of faith we cannot make which is why auditing and quality control process are needed.

Re: Bug finding is slow in spite of many eyeballs

#27
post #20

"Because in reality, many many bugs are never really found by all those given “eyeballs” in the first place. They are found when someone trips over a problem and is annoyed enough to go searching for the culprit, the reason for the malfunction." That /someone/ is able to go searching for the culprit instead of having to rely on someone ELSE to look at the source and figure out what is going on is the whole point of t…

Hazy memory, but three decades ago I read some research from the Open University in the UK that said, roughly, that the number of bugs found is a factor of the number of users.

Some apparently "bug free" programs are actually riddled with bugs but they are not found because almost nobody uses them. They are probably not fixed for the same reason ;-)

Re: Bug finding is slow in spite of many eyeballs

#28
post #24

Despite being big and different and having developed some grumpy old forum symptoms, HN is still smart. As such, I think we have a great example of memetic churn and progression here. We start with manifesto-ey essays trying to cut through some habitual thinking and paradigms. The new fundraising model. Bootstrapping. Web based SAAS is an ocean with a lot of paradigm shifts, the tech options, the business models, the…

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just god damn hard or need to be looked at with the problem in mind.

Software quality is orthogonal to the openness or closeness of the source code.

I think it is more about the will to do something great, the ability to listen to the user base and of course the technical skills of the authors.

Re: Bug finding is slow in spite of many eyeballs

#29
post #24

Despite being big and different and having developed some grumpy old forum symptoms, HN is still smart. As such, I think we have a great example of memetic churn and progression here. We start with manifesto-ey essays trying to cut through some habitual thinking and paradigms. The new fundraising model. Bootstrapping. Web based SAAS is an ocean with a lot of paradigm shifts, the tech options, the business models, the…

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just god damn hard or need to be looked at with the problem in mind. Software quality is orthogonal to the openness or closeness of the source code. I think it is more about the will to do something great, the ability to listen to the user base and of course the te…

I don't think orthogonal is quite right either.

I think it's just a matter of being complicated when you get dig into it. Open Source has certain strategies for quality that work, certain advantages (and disadvantages), different economies which means different abundances and scarcities of different resources. This "Linus’ Law" expresses something novel about the way open source projects can benefit from the advantages of being what they are. It's not an empty statement just not an airtight, always-right rule.

Re: Bug finding is slow in spite of many eyeballs

#30
post #9
post #5

Earlier quoted context omitted.

Closed source often has eyeballs specifically paid to... eyeball it.

All four combinations exist in nature: Open source code with plenty of eyeballs, open source software with no eyeballs, closed source software with tons of reviewers, and closed source software with no no reviewers. It'd be interesting to see what the percentage break down is amongst these, but it probably wouldn't be surprising. It's also another tick in the box as to why pre-checkin code review is so important - ba…

"bad code is often immortal." I have to remember to quote that!
Post reply on HN