Also of note, this blog is beautiful in the classic view.
Well, meh. It doesn't scroll when pressing the spacebar, so this is quite bad from a usability point of view.
Bug Prediction at Google
41–50 of 71 posts
Re: Bug Prediction at Google
#42Earlier quoted context omitted.
That could be true, my point was that even if a 1000 line file has a 1% chance of bug-per-line it will be marked "buggy" while a 50 line file with a 10% chance of bug-per-line won't. It makes no sense that if you took that 1000 line file and refactored it into 10 files and those 10 files had the same bugs as the original that the 1000 line file will be flagged as buggy but all 10 of the split files won't.
Not necessarily, because maybe refactoring your code to be clean enough to fit into ten files is also makes it clean enough to remove bugs. And/or hairy code doesn't end up in short files.
Who said anything about refactoring? Chop it up into 10 files verbatim. If that is indeed all it takes to cheat the algorithm, it's not very useful.
Re: Bug Prediction at Google
#43So in summary, their system highlights the files that had 'a lot of bugs lately'. While useful, this is more of a trend analysis and extrapolation rather than actual prediction. The title made me hope of something that would highlight some potential bug nests from static code analysis or the like. Interseting, but a little disappointing.
Re: Bug Prediction at Google
#44So in summary, their system highlights the files that had 'a lot of bugs lately'. While useful, this is more of a trend analysis and extrapolation rather than actual prediction. The title made me hope of something that would highlight some potential bug nests from static code analysis or the like. Interseting, but a little disappointing.
Hi. I'm Chris, the author of the blog post/work.
You're not wrong, although bug prediction is a very nebulous term in the academic literature which encompasses a pretty broad range of techniques. The term is used to properly ground the work in what's come before.
It's perhaps useful to talk a little bit about how I got where I ended up, which wasn't discussed much in the post for length reasons. I originally spent a lot of time trying to implement FixCache, a pretty complicated algorithm that looks at things such as spacial locality of bugs (bugs appear close to each other), temporal locality of bugs (bugs are introduced at the same time) and files that change together (bugs will cross cut these). It then puts it all in a LIFO cache and flags files that way. This most definitely fits in the "bug prediction" line of work, and a number of follow-up papers independently verified its success. but I was struggling deploying FixCache. This was largely because the open source code that had been written beforehand at the university lab [1] wasn't designed for MapReduce, and also because I was becoming worried that it was very difficult to understand how it flagged files, which could lead to the tool being written off as outputting false positives and then ignored.
About halfway through the internship, a pre-print of the Rahman et al. [2] paper was handed to me, which indicated that maybe everything FixCache was doing could be massively simplified and get largely the same result. We trialled both algorithms using user studies, and found that developers thought the output of the Rahman algorithm better matched their intuition as to where the hotspots in their code base was, so we went with that.
So, after this very roundabout discussion, we reach the question of "If Rahman is a subset of the FixCache algorithm, and FixCache is a bug prediction algorithm, is Rahman a bug prediction algorithm, too?" and I'll leave that for others to judge, but that's why it's been described as so :)
[1] https://github.com/SoftwareIntrospectionLab/FixCache [2] http://macbeth.cs.ucdavis.edu/fse2011.pdf
Re: Bug Prediction at Google
#45Without considering the number of submits or file length you will end up with files that are simply longer and edited more getting undue high scores. A single 100 line file will be scored to be twice as buggy as the exact same code split into two 50 line files. Considering source file lengths roughly follow a Zipfian type distribution, it seems like at least the top 5% length files would get flagged regardless unless…
Hi, I'm Chris, author of the blog post/work.
You're on the right lines, and I spent some time thinking about this. There is a couple of things that led me not to concern myself with file length. Before I start, I would note that a file just being big and getting changed isn't a problem: it has to be changed and have a bug ticket with the "bug" classification put against it.
1. As others have stated, the research does strongly point towards LOC as being part of the metric that should be looked at. [1] is a paper that discusses LOC, but it's behind the IEEE pay wall and I wasn't able to find a free version. Sorry. I'm on holiday and a bit pressed for time this morning, but I can root around for more papers if you would like. Pretty much all the models we have for bug prediction will invariably throw LOC into the pot (although as they often do PCA too, one could argue that the LOC going down could be one of the components ;) ).
2. Google, by and large, is an OOP company. The intuitive reason for why LOC leads to defects is that the file is no longer meeting the principle of single responsibility. I would hypothesize that once your file has gone past this, defects are going to start to snowball as your file is on the way to becoming what I called a Gateway file, where large amounts of functionality pass in and out of the file, even if it only does simple stuff. So not only is it going to attract defects because it's long, but it might well attract even more defects because when other functionality is modified, that file has to as well. The added problem is these files resistant to refactoring because of the possibility of breaking so much functionality.
A Very Big Philosophical Argument is whether you would put configuration files under this classification, as config files affect all sorts of functionality. Personally, I would, but most of the engineers disagreed with me, so they were left out.
3. In a very unscientific way, I did experiment with all sorts of modifiers to try and even out files that may have been "unfairly" flagged due to some metric like LOC or number of commits or whatever. I eventually abandoned this, not just because I found the results largely unsatisfying, but I realized I really was doing things in an unscientific manner without any real justification for why I was doing it. Without a sound justification for evening it out, which LOC doesn't have, I decided not to do it.
Hope this helps a little bit. I very much see your point, and a number of engineers brought this up during user studies, but those big files that were getting flagged really were problems.
[1] http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4297...
Re: Bug Prediction at Google
#46Earlier quoted context omitted.
Not necessarily, because maybe refactoring your code to be clean enough to fit into ten files is also makes it clean enough to remove bugs. And/or hairy code doesn't end up in short files.
Who said anything about refactoring? Chop it up into 10 files verbatim. If that is indeed all it takes to cheat the algorithm, it's not very useful.
I'm Chris, author of the post/work.
Cheating the algorithm is indeed a possibility and is actually very easy:
1. Just don't file bug tickets.
2. If you do, don't attach the tickets to changes.
3. If you attach the tickets to changes, don't put the "bug" type on the ticket.
Doing things to deliberately cheat the algorithm is likely not going to pass code review.
One common misconception is that the algorithm is a stick to beat developers with. It really, really isn't, and I go to great lengths to try and make this clear in the internal docs (although the blog post doesn't do this as much). The idea is just to provide another insight into the code, so that devs don't accidentally change something that's a real problem without help.
The algorithm is just a tool to help developers understand their code. It's not subjective because different teams use the bug tracker differently, so a score from a file in one project cannot be compared to a file from another project. We mitigate this by only taking off the top 10% of code across the entire codebase, so hopefully we by and large only flag the worst offenders. Part of what I wanted to do was to set up a web app so teams could claim various parts of the code base, and then run the algorithm on a team-by-team basis instead, but I ran out of time on that.
I think this is actually one of the nice things about the algorithm, because I don't like the idea of implicitly pitting people or teams against one another. I just like that people can go "Oh hey, this has been a real problem for us, let's tread carefully here."
Re: Bug Prediction at Google
#47https://github.com/igrigorik/bugspots - for anyone that's curious to try it on their own git repo. (improvements welcome! :-))
This is so awesome, thanks igrigorik :)
Re: Bug Prediction at Google
#48Trouble is, lots of nontrivial bugs are not module (or file) specific. The occur due to interaction between modules. Implicit assumptions on input are violated, invalid output is produced and continues propagating in the system.
[Hope this technology will help make Chrome more stable. Do they even consider resource leaks bugs?]
Re: Bug Prediction at Google
#49https://github.com/igrigorik/bugspots - for anyone that's curious to try it on their own git repo. (improvements welcome! :-))
I came to the comments hoping to find that this existed!
Re: Bug Prediction at Google
#50https://github.com/igrigorik/bugspots - for anyone that's curious to try it on their own git repo. (improvements welcome! :-))
looks good, but you might move the pattern matching in scanner.rb out to a config file so users can search for other keywords.