They haven't actually analyzed 100M bugs, they've analyzed a list of bug reports. They haven't analyzed how quickly bugs are resolved, they've analyzed how quickly bugs are marked as resolved. The distinction is important. Nowhere in the report there's any attempt to judge the quality of the data and its reliability. Oh, and the honest answer to the question "Why did we create this report?" is probably PR.
- You’re absolutely right, not all the 100M bug reports are actually bugs. However, we highlighted this under the "Time to Close" section: "These are most likely not programmatic bugs, but could be support issues or spam." How do you think we should highlight this more to avoid confusion? - About bug resolution time, Instabug is used by many companies as their main bug reporting tool or they forward these bugs to ano…
I think that this report is only useful in showing app developers that the patterns they encounter in their bug reports are common in the entire ecosystem, not special to their specific app requiring further investigation. Keep the patterns, keep the information about integration with external tools (customers might find it useful). Scraps the rest.
The main problem in the report is that you try to answer questions which your data and analysis is inherently incapable of answering. For example:
- "Which manufacturers have the most bugs?" - "Which UI orientation has more issues?" - "Which locale has the most bugs?" - "How does battery affect app stability?" - "Which OS has buggier apps?"
As other commenters have mentioned, your results could be just artifacts of the user demographics (or any number of other confounders). The answers are, at best, meaningless.
There are significant inconsistencies in figures 1 and 2. They definitely do not agree with "Errors discovered through Instabug are most likely to be resolved within 24 hours of being reported." (except in the narrow technical sense of the first day being the most likely day).
Even if the data was sufficient, there's no mention of statistical significance in comparisons. For example, Danish is the locale with the most bugs per user. However, you have quite a lot of locales and random variability is expected. Is the difference statistically significant?