Bug finding is slow in spite of many eyeballs • Hacker News Board

Re: Bug finding is slow in spite of many eyeballs

#61

Post by cheatsheet » Mon, Feb 23, 2015, 7:06 PM UTC

fdej wrote: post #60

cheatsheet wrote: post #56

Earlier quoted context omitted.

Sure, but there will still be bugs. We might have to figure out what a bug is semantically on the next layer of abstraction as we are developing it (and have the capacity to formally describe it), but there will be opinions, and thus there will be bugs.

Of course. In the end, we don't even have a perfect understanding of the physics on which digital technology is built. Nonetheless, we can evidently do much better than we do today, and delegate more of the hard work to our robotic slaves.

I think a wise person would hesitate at judging a task hard or easy.

Re: Bug finding is slow in spite of many eyeballs

#62

Post by jacquesm » Mon, Feb 23, 2015, 7:25 PM UTC

lmm wrote: post #38

shin_lao wrote: post #28

Earlier quoted context omitted.

Anyone who has a couple of year into software engineering knows that there is diminishing return over the number of "eye balls" and that some problems are just god damn hard or need to be looked at with the problem in mind. Software quality is orthogonal to the openness or closeness of the source code. I think it is more about the will to do something great, the ability to listen to the user base and of course the te…

The article shows it's not about that > Perhaps you think these 30 bugs are really tricky, deeply hidden and complicated logic monsters that would explain the time they took to get found? Nope, I would say that every single one of them are pretty obvious once you spot them and none of them take a very long time for a reviewer to understand. For me the takeaway is that even something as basic and mature as curl can't…

It's the devil that we know. New environments: new exploit classes.

Re: Bug finding is slow in spite of many eyeballs

#63

Post by lgunsch » Mon, Feb 23, 2015, 7:37 PM UTC

fdej wrote: post #49

mcdoug wrote: post #48

Earlier quoted context omitted.

I feel like if we did this we'd never get to an actual working system. How do you formally verify that a remote file system interacting with a faulty hard disk is doing the right thing? This reminds me of Hurd, which is arguably a superior design that will never be finished.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic. Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the fu…

>we need tools that make development easier, to allow keeping up with the rapidly changing requirements in the software world.

The practice of Test driven development is a great solution to this. The problem is very few people use it, many people actually are against it, and many of those that do do it, do it incorrectly (not on purpose). I would think that small chunks of singly-purposed functionality would be much easier to verify.

Re: Bug finding is slow in spite of many eyeballs

#64

Post by erikb » Mon, Feb 23, 2015, 8:06 PM UTC

Well the conclusion is not surprising. Bugs aren't found because many people "read" the source code, but many people of many different skills use it and therefore every problem that is hard to you will some day find a person that has the specific domain knowledge to fix it much more easily.

Also I'm a little surprised at the size of the dataset and the choices, given that open source probably fixes an easier bug faster than a higher priority one.

Re: Bug finding is slow in spite of many eyeballs

#65

Post by maemre » Mon, Feb 23, 2015, 10:21 PM UTC

jdc wrote: post #57

fdej wrote: post #49

Earlier quoted context omitted.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic. Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the fu…

What are some examples of non-trivial working formally verified software?

There is Quark [1], a web browser with formally verified kernel. Here, kernel is a process which manages other slave (helper) processes which render the page actually.

[1]: http://goto.ucsd.edu/quark/

Re: Bug finding is slow in spite of many eyeballs

#66

Post by schoen » Mon, Feb 23, 2015, 11:27 PM UTC

lgunsch wrote: post #63

fdej wrote: post #49

Earlier quoted context omitted.

That's certainly an issue, and there's no easy way around the fact that even formally specifying the behavior of a program that has to interact with the outside world is problematic. Nonetheless, some of the academic research that has been done on formal verification is quite impressive (including the development of actual nontrivial working software). And it can be done piecemeal -- you could formally certify the fu…

>we need tools that make development easier, to allow keeping up with the rapidly changing requirements in the software world. The practice of Test driven development is a great solution to this. The problem is very few people use it, many people actually are against it, and many of those that do do it, do it incorrectly (not on purpose). I would think that small chunks of singly-purposed functionality would be much…

There's kind of a considerable gap between formally verified software and test-driven development, though. For example, a parser or the routines that go into it might pass dozens or hundreds of unit tests, yet still fail to parse certain inputs correctly, fail to parse them in exactly the same way as another parser (as Langsec researchers have emphasized, often producing possible time-of-check/time-of-use vulnerabilities), or even have a memory corruption bug.

I don't mention this in order to criticize test-driven development or the improvements it can bring to software reliability or safety, just to point out that there's still a big gap from there to a formal proof of correctness.

Re: Bug finding is slow in spite of many eyeballs

#67

Post by schoen » Mon, Feb 23, 2015, 11:30 PM UTC

jacquesm wrote: post #62

lmm wrote: post #38

Earlier quoted context omitted.

The article shows it's not about that > Perhaps you think these 30 bugs are really tricky, deeply hidden and complicated logic monsters that would explain the time they took to get found? Nope, I would say that every single one of them are pretty obvious once you spot them and none of them take a very long time for a reviewer to understand. For me the takeaway is that even something as basic and mature as curl can't…

It's the devil that we know. New environments: new exploit classes.

There are some great examples of that (return-oriented programming as a way of getting around non-executable data segments), but surely there are some tools where adopting them was a pure win for safety and correctness over the status quo.

Re: Bug finding is slow in spite of many eyeballs

#68

Post by DonHopkins » Mon, Feb 23, 2015, 11:45 PM UTC

To quote Theo De Raadt:

My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code.

Re: Bug finding is slow in spite of many eyeballs

#69

Post by br3w5 » Mon, Mar 02, 2015, 8:24 PM UTC

t_7u_ol8 wrote: post #39

br3w5 wrote: post #12

Earlier quoted context omitted.

But just because they're paid to do it doesn't mean they care, or that they're any good at it.

You can apply that attitude in general to open source software as well. Most people using open source software trust it as implicitly as they would have to trust closed source software, they have to, because reviewing and comprehending all of the code running on any typical machine is impossible. Most people simply use open source and don't care about (or aren't capable of) dealing with the code, and of the ones that…

Totally true - i think if you want to use open source you should be prepared to read the code in order to understand it and for debugging