Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug Prediction at Google

google-engtools.blogspot.com

Post reply on HN
Print view View original

21–30 of 71 posts

Re: Bug Prediction at Google

#21

Post by jdp23 »

groby_b wrote: post #9
po wrote: post #6

Earlier quoted context omitted.

I was thinking the same thing. I was expecting them to analyze the diffs and keep a cache of code blocks that are changing in each bugfix. Keeping track of bugs at the class/function/method level would seem to be more powerful.

Existing research points to "number of bug-fixing commits" as being enough to predict. I would assume that's due to the fact that bugs are not always constrained to a single function, but usually to closely related code. Which often means all that related code is in the same file.

Indeed. Also, typically code "ownership" is at the file level -- and that's likely to correlate strongly with defects in situations where a peron who hasn't been well trained (or is just a bad programmer) is writing or maintaining the code.

Re: Bug Prediction at Google

#22

Post by jdp23 »

T_S_ wrote: post #5

I've got a great bug prediction tool: ghc, the haskell compiler. Whenever I use a value of the wrong type my program won't compile. In my code it is always a bug when that happens. (Some smarter people can write non-buggy code the compiler doesn't like.) Sometimes I can trick ghc into using the wrong type. Like when I use "String" to mean "File". That's when all the bugs show up as run-time errors. But I should know…

Defect detection != defect prediction

Re: Bug Prediction at Google

#23

Post by nl »

_delirium wrote: post #16
nl wrote: post #14

Earlier quoted context omitted.

I'm not sure what your point is. Yes, type systems can catch a class of bugs. Google - like most of the software development world - has a lot of people who believe in type systems completely. That's why they have tools like the Closure JS compiler[1], which provides type-checking for Javascript, and GWT[2], which produces (un-typesafe) Javascript/CSS/HTML from (mostly)typesafe Java. Using methods of reducing bugs (s…

Your last statement's the part I disagree with, though it depends on what you mean by "orthogonal". If you mean that they're two different approaches that can coexist, then yes. But I don't think they target orthogonal classes of bugs. In both cases, the goal is to employ some algorithmic, decidable method at compile-time to predict whether a given piece of code is "correct" or "incorrect", trading off the possibilit…

>The ambition, at least, of static type systems is to render accurate bug-prediction impossible, because every bug statically predictable at compile-time will be a type error.

I might not use Haskell, but I do understand what a type system does, and why it is important.

BUT, not all bugs are statically predictable at compile-time. Take things like cross-browser compatibility - something like GWT goes a long way to reducing bugs with that, but no type system will protect you from a new bug in a new browser you need to work around.

(Edit: by orthogonal I meant "statistically independent". Given a piece of code written in a type safe language, this method will predict bugs independently of a type system.)

Re: Bug Prediction at Google

#24

Post by jdp23 »

_delirium wrote: post #16
nl wrote: post #14

Earlier quoted context omitted.

I'm not sure what your point is. Yes, type systems can catch a class of bugs. Google - like most of the software development world - has a lot of people who believe in type systems completely. That's why they have tools like the Closure JS compiler[1], which provides type-checking for Javascript, and GWT[2], which produces (un-typesafe) Javascript/CSS/HTML from (mostly)typesafe Java. Using methods of reducing bugs (s…

Your last statement's the part I disagree with, though it depends on what you mean by "orthogonal". If you mean that they're two different approaches that can coexist, then yes. But I don't think they target orthogonal classes of bugs. In both cases, the goal is to employ some algorithmic, decidable method at compile-time to predict whether a given piece of code is "correct" or "incorrect", trading off the possibilit…

I think this misunderstands the goals of bug prediction, which are typically along the lines of

- directing engineering resources to where they're likely to be most leveraged from a quality standpoint [what Google focused on]

- estimating how many resources to devote to bug fixing and/or long it will take an in-progress code base to stabilize [the goal of a lot of the work Micorosft did with Windows in the early 2000s]

- and estimating post-release defects to estimate (and perhaps direct) customer support and maintenance resources [more typical in the hardware world than software]

And I wouldn't say that static type systems have an ambition of rendering bug-prediction impossible. If you can truly detect all errors at compile time, then prediction's trivial: no bugs remain!

Re: Bug Prediction at Google

#26

Post by adanto6840 »

Any chance we can get Github to implement this on commits or add it as an optional toggle? It just makes so much sense...

As a tech manager the ability to know when you should dedicate just a bit more time to a given file would be fantastic, so much so that I'd probably even pay a small additional fee just for this functionality!

Would definitely love to see some implementation of this for Git though, may even work on a little script for it if no one beats me to it... :-)

Re: Bug Prediction at Google

#29

Post by jacques_chester »

What I like about Google is that they take uncontroversial "should-dos" -- you should unit test, you should review, you should pay more attention to bug-prone models -- and then build and improve tools that supports those practices.

To me one of the reasons agile so swept our field was because it rode the back of the increasing availability of two genres of toolset: the unit testing framework and the bug/issue/story/feature tracker.

Re: Bug Prediction at Google

#30

Post by esrauch »

tesseract wrote: post #8
esrauch wrote: post #2

Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files. Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless…

I've seen a few studies (sorry, no citations handy, if I have time I'll revisit this comment) which concluded that above some fairly low floor, bugginess is strongly positively correlated with file length. Which is to say, always flagging the top 5% longest files as being among the buggiest has a good chance of being the right thing to do.

That could be true, my point was that even if a 1000 line file has a 1% chance of bug-per-line it will be marked "buggy" while a 50 line file with a 10% chance of bug-per-line won't.

It makes no sense that if you took that 1000 line file and refactored it into 10 files and those 10 files had the same bugs as the original that the 1000 line file will be flagged as buggy but all 10 of the split files won't.

Post reply on HN