Scaling Static Analyses at Facebook
m-cacm.acm.org
Scaling Static Analyses at Facebook
1–10 of 27 posts
Re: Scaling Static Analyses at Facebook
#2> Zoncolan catches more SEVs than either manual security reviews or bug bounty reports. We measured that 43.3% of the severe security bugs are detected via Zoncolan. At press time, Zoncolan's "action rate" is above 80% and we observed about 11 "missed bugs."
>. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of both mobile (Android and Objective C) code and backend C++ code
> All codebases see thousands of code modifications each day and our tools run on each code change. For Zoncolan, this can amount to analyzing one trillion lines of code (LOC) per day.
11 "missed bugs" on the 100 mm server-side lines of code per run, or ever?
Re: Scaling Static Analyses at Facebook
#3> Zoncolan catches more SEVs than either manual security reviews or bug bounty reports. We measured that 43.3% of the severe security bugs are detected via Zoncolan. At press time, Zoncolan's "action rate" is above 80% and we observed about 11 "missed bugs." >. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of…
Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.
Re: Scaling Static Analyses at Facebook
#4Always cool to read about scale.
Re: Scaling Static Analyses at Facebook
#5> Zoncolan catches more SEVs than either manual security reviews or bug bounty reports. We measured that 43.3% of the severe security bugs are detected via Zoncolan. At press time, Zoncolan's "action rate" is above 80% and we observed about 11 "missed bugs." >. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of…
Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.
Yes, that's the main culprit with traditional static analysis. No one wants to review the results, because the amount of signal to noise is far too low. And also since it's an optional thing and not enforced by the compiler.
I think this is where languages with stronger inbuilt analysis (e.g. Rust) win: The results are better, and since the analysis is always running as part of a compiler pass there are no huge jumps in indicated bugs at once (like what would happen if one would run Coverity on a legacy C++ codebase).
Re: Scaling Static Analyses at Facebook
#6Earlier quoted context omitted.
Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.
Yes, that's the main culprit with traditional static analysis. No one wants to review the results, because the amount of signal to noise is far too low. And also since it's an optional thing and not enforced by the compiler. I think this is where languages with stronger inbuilt analysis (e.g. Rust) win: The results are better, and since the analysis is always running as part of a compiler pass there are no huge jumps…
This is less true of more advanced static analysis tools.
I mean, ultimately we agree. Most people don't trust static analysis tools because they have had bad experiences with them. I just suspect most people should try them again. The state of the art is quite good in that space.
Re: Scaling Static Analyses at Facebook
#7> Zoncolan catches more SEVs than either manual security reviews or bug bounty reports. We measured that 43.3% of the severe security bugs are detected via Zoncolan. At press time, Zoncolan's "action rate" is above 80% and we observed about 11 "missed bugs." >. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of…
It's explained in the article:
> We also use the traditional security programs to measure missed bugs (that is, the vulnerabilities for which there is a Zoncolan category), but the tool failed to report them. To date, we have had about 11 missed bugs, some of them caused by a bug in the tool or incomplete modeling.
A missed bug is presumably one that the tool is designed to spot, but which it didn't during the period in which it has been running.
Re: Scaling Static Analyses at Facebook
#8> Zoncolan catches more SEVs than either manual security reviews or bug bounty reports. We measured that 43.3% of the severe security bugs are detected via Zoncolan. At press time, Zoncolan's "action rate" is above 80% and we observed about 11 "missed bugs." >. For the server-side, we have over 100-million lines of Hack code, which Zoncolan can process in less than 30 minutes. Additionally, we have 10s of millions of…
Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.
It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.
Re: Scaling Static Analyses at Facebook
#9Earlier quoted context omitted.
Also, the main issue with static analysis tools tends to be not false negatives, but false positives. That is, they churn out tons and tons of alerts that aren't actually bugs. Some such systems alert so much that they aren't worth using.
It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.
I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but when it does find something it's usually a legit issue.
But the balance of deep analysis and low false positives remains elusive. I'd be really stunned if FB really achieved a breakthrough in this area.
I do want to be wrong about this.
Re: Scaling Static Analyses at Facebook
#10Earlier quoted context omitted.
It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.
I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but…
In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.