Bug Prediction at Google

#1

Post by nl » Thu, Dec 15, 2011, 1:14 AM UTC

Bug Prediction at Google
google-engtools.blogspot.com

Re: Bug Prediction at Google

#2

Post by esrauch » Thu, Dec 15, 2011, 2:10 AM UTC

Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files.

Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless the bug distribution is extremely uneven.

Re: Bug Prediction at Google

#3

Post by jdp23 » Thu, Dec 15, 2011, 2:24 AM UTC

If you're into defect prediction, it's worth checking out the work the Empirical Software Engineering group at Microsoft Research has done over the years -- http://research.microsoft.com/en-us/groups/ese/publications....

Re: Bug Prediction at Google

#4

Post by eric-hu » Thu, Dec 15, 2011, 2:38 AM UTC

I would gladly use this tool and provide feedback if it were released as something that tacks onto git.

Re: Bug Prediction at Google

#5

Post by T_S_ » Thu, Dec 15, 2011, 2:38 AM UTC

I've got a great bug prediction tool: ghc, the haskell compiler. Whenever I use a value of the wrong type my program won't compile. In my code it is always a bug when that happens. (Some smarter people can write non-buggy code the compiler doesn't like.)

Sometimes I can trick ghc into using the wrong type. Like when I use "String" to mean "File". That's when all the bugs show up as run-time errors. But I should know a string is not a file. And hopefully somebody writes a module that knows what a "File" really is and stops me from writing buggy code if I use it.

Re: Bug Prediction at Google

#6

Post by po » Thu, Dec 15, 2011, 2:46 AM UTC

esrauch wrote: post #2

Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files. Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless…

I was thinking the same thing. I was expecting them to analyze the diffs and keep a cache of code blocks that are changing in each bugfix. Keeping track of bugs at the class/function/method level would seem to be more powerful.

Re: Bug Prediction at Google

#7

Post by nl » Thu, Dec 15, 2011, 2:57 AM UTC

T_S_ wrote: post #5

I've got a great bug prediction tool: ghc, the haskell compiler. Whenever I use a value of the wrong type my program won't compile. In my code it is always a bug when that happens. (Some smarter people can write non-buggy code the compiler doesn't like.) Sometimes I can trick ghc into using the wrong type. Like when I use "String" to mean "File". That's when all the bugs show up as run-time errors. But I should know…

Type errors are only one type of bug.

Re: Bug Prediction at Google

#8

Post by tesseract » Thu, Dec 15, 2011, 3:03 AM UTC

esrauch wrote: post #2

Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files. Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless…

I've seen a few studies (sorry, no citations handy, if I have time I'll revisit this comment) which concluded that above some fairly low floor, bugginess is strongly positively correlated with file length.

Which is to say, always flagging the top 5% longest files as being among the buggiest has a good chance of being the right thing to do.

Re: Bug Prediction at Google

#9

Post by groby_b » Thu, Dec 15, 2011, 3:06 AM UTC

po wrote: post #6

esrauch wrote: post #2

Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files. Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless…

I was thinking the same thing. I was expecting them to analyze the diffs and keep a cache of code blocks that are changing in each bugfix. Keeping track of bugs at the class/function/method level would seem to be more powerful.

Existing research points to "number of bug-fixing commits" as being enough to predict.

I would assume that's due to the fact that bugs are not always constrained to a single function, but usually to closely related code. Which often means all that related code is in the same file.

Re: Bug Prediction at Google

#10

Post by _delirium » Thu, Dec 15, 2011, 3:11 AM UTC

nl wrote: post #7

T_S_ wrote: post #5

I've got a great bug prediction tool: ghc, the haskell compiler. Whenever I use a value of the wrong type my program won't compile. In my code it is always a bug when that happens. (Some smarter people can write non-buggy code the compiler doesn't like.) Sometimes I can trick ghc into using the wrong type. Like when I use "String" to mean "File". That's when all the bugs show up as run-time errors. But I should know…

Type errors are only one type of bug.

True, but the whole point of designing modern type systems is to attempt to catch as many common bugs in the static-check phase as possible (while still being decidable), rather than purely checking datatypes (int, string, etc.) in the classical sense. I don't think it's too unreasonable to compare it to empirical bug prediction: they're a rationalist vs. an empiricist approach to statistically predicting where bugs probably lie, with some various tradeoffs.