Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Static Analyses at Facebook

m-cacm.acm.org

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Scaling Static Analyses at Facebook

#21

Post by wallstprog »

I think one of the unstated problems with static analysis is just keeping track of the results. I know that when I started working with these tools, it was a huge PITA just dealing with the various output files.

That's why I created tools to convert the output from different tools into a common CSV format that can be databased and used to compare output from different tools, or from different versions of the code (e.g., after fixing errors reported by the tools).

These tools currently work with cppcheck, clang and PVS-Studio and can be found here: http://btorpey.github.io/blog/categories/static-analysis/

Re: Scaling Static Analyses at Facebook

#22

Post by nh_99 »

wallstprog wrote: post #21

I think one of the unstated problems with static analysis is just keeping track of the results. I know that when I started working with these tools, it was a huge PITA just dealing with the various output files. That's why I created tools to convert the output from different tools into a common CSV format that can be databased and used to compare output from different tools, or from different versions of the code (e.…

Interesting approach. Where I work, we use Jenkins for collecting results. That way, for each build of our application we have a history of results for static analysis. Jenkins has good tools for storing and displaying this information, as well as the ability to show trends over time.

Re: Scaling Static Analyses at Facebook

#23

Post by nickpsecurity »

One of the things I like about this article is that it gives another example showing how formal methods catches deep errors unlikely to be caught with human review or testing:

"Overall, the error trace found by Infer has 61 steps, and the source of null, the call to X509 _ gmtime _ adj () goes five procedures deep and it eventually encounters a return of null at call-depth 4. "

I think the example Amazon gave for TLA+ was thirty-something steps. Most people's minds simply can't track 61 steps into software. Tests always have a coverage issue.

Re: Scaling Static Analyses at Facebook

#24

Post by wallstprog »

nh_99 wrote: post #22
wallstprog wrote: post #21

I think one of the unstated problems with static analysis is just keeping track of the results. I know that when I started working with these tools, it was a huge PITA just dealing with the various output files. That's why I created tools to convert the output from different tools into a common CSV format that can be databased and used to compare output from different tools, or from different versions of the code (e.…

Interesting approach. Where I work, we use Jenkins for collecting results. That way, for each build of our application we have a history of results for static analysis. Jenkins has good tools for storing and displaying this information, as well as the ability to show trends over time.

If Jenkins works for you, great. It does seem to support both clang and cppcheck, although not PVS-Studio (which is one of the better tools out there in my experience).

Personally, I'm happier with plain old text files that can be manipulated with awk, grep, etc., can be databased if needed (since they're csv files) -- and can also be compared using my all-time favorite software, Beyond Compare. (http://btorpey.github.io/blog/2013/01/29/beyond-compare/).

Re: Scaling Static Analyses at Facebook

#25

Post by j88439h84 »

clarus wrote: post #18
j88439h84 wrote: post #10

Earlier quoted context omitted.

In python, Pylint and mypy find real bugs all the time, plenty of false positives but still very usable.

Pylint and mypy are about syntax and type-checking. While I agree these kind of tools work well, if think that by static analysis people usually imply something which goes farther than that. For example, the languages Facebook is citing (C++, Java, ...) already include type-checking in the compiler.

Pylint catches lots of semantic bugs, far from just syntax.

Re: Scaling Static Analyses at Facebook

#26

Post by nickpsecurity »

m0zg wrote: post #9
muglug wrote: post #8

Earlier quoted context omitted.

It sounds (from the article) like they have some sort of heuristic for determining potential severity, and they're ok with more false-positives in areas where the potential damage from a false-negative is very high.

I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but…

You kind of explained it yourself saying 15-17 years ago. Lots of things can improve in that time-frame. PreFix wasn't even that focused on reducing false positives IIRC. Some today are focused on keeping false positives down. A few benchmarked are here in this also-biased article:

https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benc...

I bring them up because they made the open-source K Framework and a C semantics. Another commenter says PVS-Studio is pretty good. Since Synopsis owns Coverity now, I'd recommend RV-Match (little to no false positives) followed by PVS-Studio.

Re: Scaling Static Analyses at Facebook

#27

Post by m0zg »

nickpsecurity wrote: post #26
m0zg wrote: post #9

Earlier quoted context omitted.

I might be biased, but I've never seen these systems work well in practice. Some 15-17 years ago Microsoft depoloyed a system called PreFix which would find genuine, hard to find bugs, but then bury them under a mountain of false positives, so few teams ran it, and even fewer looked at the results. I like what LLVM did in this area. Its SCA is not very comprehensive (so it can't be relied upon for deep analysis), but…

You kind of explained it yourself saying 15-17 years ago. Lots of things can improve in that time-frame. PreFix wasn't even that focused on reducing false positives IIRC. Some today are focused on keeping false positives down. A few benchmarked are here in this also-biased article: https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benc... I bring them up because they made the open-source K Framework and a C sem…

Don't know about Coverity, but a customer did a Fortify scan on our code last year (as a part of acceptance), and it didn't find any issues. Which I find really hard to believe that there aren't any issues, seeing that the codebase is 200KLOC of pretty gnarly C++ with probably another 500KLOC in source-level C++ deps (we compile from source where we can). Either the team was comprised entirely of gods of C++ programming, or it doesn't really "fortify" all that much. I'll let you decide which one is more probable.
Post reply on HN