Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What We Learned from Analyzing 100M Bugs

instabug.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: What We Learned from Analyzing 100M Bugs

#21

Post by arriu »

oldgradstudent wrote: post #15

They haven't actually analyzed 100M bugs, they've analyzed a list of bug reports. They haven't analyzed how quickly bugs are resolved, they've analyzed how quickly bugs are marked as resolved. The distinction is important. Nowhere in the report there's any attempt to judge the quality of the data and its reliability. Oh, and the honest answer to the question "Why did we create this report?" is probably PR.

Agreed, I was excited to find out whether data about the bugs could have been used to predict future bugs.

Re: What We Learned from Analyzing 100M Bugs

#22

Post by barbegal »

bradjohnson wrote: post #4

I think the y axis: "% Bugs" in Fig. 1 has an incorrect scale. It doesn't add to 100%. Also, it doesn't seem consistent with the claim: "Bugs discovered through Instabug are most likely to be resolved within 24 hours of being reported"

Yeah going by that graph it appears that ~1.5% of bugs are fixed in 24 hours, ~5% within a week, ~10% within 30 days and only ~13% of bugs are fixed at all. That leaves 87% of bugs still to be resolved.

From the graph of total bugs vs time. It appears like at the current time ~10% of all bugs have been reported in the last 30 days. Even if all those bugs were magically fixed tomorrow, that would only be ~20% of bugs within 30 days so we can claim:

"Bugs discovered through Instabug are unlikely to be resolved within 30 days" and "1.5% of bugs discovered through Instabug are likely to be resolved within 24 hours of being reported"

Re: What We Learned from Analyzing 100M Bugs

#23

Post by aprilledaughn »

oldgradstudent wrote: post #15

They haven't actually analyzed 100M bugs, they've analyzed a list of bug reports. They haven't analyzed how quickly bugs are resolved, they've analyzed how quickly bugs are marked as resolved. The distinction is important. Nowhere in the report there's any attempt to judge the quality of the data and its reliability. Oh, and the honest answer to the question "Why did we create this report?" is probably PR.

- You’re absolutely right, not all the 100M bug reports are actually bugs. However, we highlighted this under the "Time to Close" section: "These are most likely not programmatic bugs, but could be support issues or spam." How do you think we should highlight this more to avoid confusion?

- About bug resolution time, Instabug is used by many companies as their main bug reporting tool or they forward these bugs to another bug tracker like Jira and we have a two-way sync so whenever it gets resolved over there, it’s resolved at Instabug as well. That’s why we used the word "resolved" not "fixed" because each company has their own definition. I hope this makes sense.

- About the quality and reliability of the data: Oh, we didn’t mean to be protective about this! On the contrary, we’d love to get your feedback. What would you like to know?

- About your third point, I respectfully disagree. As the person who spent the most hours working on this report, I can tell you honestly that it was not for PR. We just wanted to put something out there that would hopefully be valuable to the people in our community. We initially shared this with our own users for them to have benchmarks. This is the first time we’ve released anything like it, so it was an experiment for us to be honest and I’m loving all these comments because it helps us know what to do better next time around.

Re: What We Learned from Analyzing 100M Bugs

#24

Post by okgabr »

coldtea wrote: post #16

I expected something much more interesting: e.g. most common types of bugs, or causes of bugs -- and thus suggestions on how they could be avoided.

Good point and sorry to disappoint you! The good news is that we still have a lot to share, this is the first time in six years to dig deeper into our data and share it with the community. I’m sure we’ll do more and more soon. A series about the most common causes of bugs and suggestions on how they could be avoided would definitely be a great start!

Re: What We Learned from Analyzing 100M Bugs

#25

Post by aprilledaughn »

arriu wrote: post #21
oldgradstudent wrote: post #15

They haven't actually analyzed 100M bugs, they've analyzed a list of bug reports. They haven't analyzed how quickly bugs are resolved, they've analyzed how quickly bugs are marked as resolved. The distinction is important. Nowhere in the report there's any attempt to judge the quality of the data and its reliability. Oh, and the honest answer to the question "Why did we create this report?" is probably PR.

Agreed, I was excited to find out whether data about the bugs could have been used to predict future bugs.

Is there anything specific you'd like to know more about? It would be great to understand what kind of info people are looking for to know what to publish in the future.

Re: What We Learned from Analyzing 100M Bugs

#26

Post by chatmasta »

aprilledaughn wrote: post #3
fhood wrote: post #2

Hmm, all this is basically what I would expect....wait..."Most bugs are reported from iPhones, while more bugs/user are reported from LG devices." ....Why LG?

Thanks for checking out the report! I'm part of the team who put it together :) Yeah, we thought LG was interesting too. When it comes to Android, we expected Samsung to take first place tbh, but we found more bugs/user reported from LG and Google devices (Fig. 9). This could be explained by our technical user base and the popularity of Nexus devices with Android developers. So the higher proportion of bugs/user we s…

Or maybe it’s the opposite, i.e. devs are not testing on LG devices so they miss corner cases and ship bugs to them?

Re: What We Learned from Analyzing 100M Bugs

#28

Post by aprilledaughn »

chatmasta wrote: post #26
aprilledaughn wrote: post #3

Earlier quoted context omitted.

Thanks for checking out the report! I'm part of the team who put it together :) Yeah, we thought LG was interesting too. When it comes to Android, we expected Samsung to take first place tbh, but we found more bugs/user reported from LG and Google devices (Fig. 9). This could be explained by our technical user base and the popularity of Nexus devices with Android developers. So the higher proportion of bugs/user we s…

Or maybe it’s the opposite, i.e. devs are not testing on LG devices so they miss corner cases and ship bugs to them?

Interesting! Could be. Our analysis is based on what we know about our users' behavior but certainly not definitive. All the data here is open to interpretation.
Post reply on HN