Live data from Hacker News

Welcoming Semmle to GitHub

github.blog

91–100 of 110 posts

Re: Welcoming Semmle to GitHub

#91

Interesting to see the differences between Github and Gitlab's strategy in this arena. Github appears to be going the aqui-hire route with Semmle, dependabot, pullpanda etc, where as I don't think Gitlab's made an acquisition for a year or two.

GitLab published what they're interested in: https://about.gitlab.com/handbook/acquisitions/ . It's an amazing, one-of-a-kind doc. One of their constraints ( https://about.gitlab.com/handbook/acquisitions/#what-we-offe... ) is quite limiting, though: > The total purchase price of the deal, paid in cash, will not exceed $1M and will be the total and only compensation for the entire deal.

It looks like they're buying (big) features, not complete solutions or companies. That's actually an interesting approach; I'm sure others do that, but maybe not as explicitly.

It would allow a small team of hackers to have a decent exit without having to go through the whole startup road.

Re: Welcoming Semmle to GitHub

#92

Earlier quoted context omitted.

Welp, guess it's time to bust out the "I'm offended and we need to change this lingo" card because corporations aren't people and we should stop referring to them that way

I mean, in all seriousness, aren't they a little bit like people? https://www.npr.org/2014/07/28/335288388/when-did-companies-...

They are slow AIs: https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future

Re: Welcoming Semmle to GitHub

#93
post #86

I've just tested their lgtm.com on our codebase: 1) identified str.replace('[ABC]+', '') correctly as a bug (looks like a regex but is string literal) 2) identified various unnecessary code that TypeScript overlooked 3) identified double-unescaping of html (this one would have probably gone unnoticed for years) And a bunch of other stuff. No actual vulnerability in our case, but still very useful. I'm enabling their…

Tested, Kotlin is not supported, nor is Swift.

Re: Welcoming Semmle to GitHub

#94
post #76

Earlier quoted context omitted.

It’s absolutely useless in the wrong hands, so I don’t think you’d necessarily get a good signal by asking your average blue teamer. It’s a godsend for someone who spends a lot of time auditing code and has some experience writing code analysis tools. I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you li…

I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.

Anecdata, I have also successfully used it to unearth complicated bugs that I already knew must exist, but didn’t have a good way to find.

Think of a query like: Find all calls to function A that have an output pointer-pointer of type B in the last argument position and also have a boolean return type, verify the input dereferenced B is NULL, then verify that iff A assigns output to B that the return type is false and that the calling frame also includes a later call to function C to clean up B. This type of thing run on millions of lines of code.

This can get as crazy as you want it to be and did work, but there is a “but.” The query language is so verbose and powerful, there were many, many ways to represent the same query that all had drastically different performance profiles. The docs were woefully underwhelming in that regard — they simply stated what a member did, but not anything at all related to memory or performance implications. That, coupled with the fact that nearly every complex query during dev hit the wall of the JVM killing it for taking too much time/mem, it became apparent that they must have tooling for the tooling to analyze performance and memory profiles of queries (not unlike all the effort put into SQL over the years). Also, no debugger; resorted to “datalog printf debugging” (ie, including internal clauses as outputs and chopping off later parts of the query...)

I spent a few weeks only writing queries and in every non-trivial one I needed a senior person there to review/rearrange a few clauses to go from e.g. 30 minute runtime to 15 seconds. That left me with a feeling that I was constantly fighting the docs and lack of tooling and would constantly need their help to tune things. Nothing was fundamentally wrong with the queries — it was just not documented anywhere that some filtering should be done caller->down vs. other kinds of checks in the same query should be callee->up.

I had questions regarding scaling up to 1B+ LoC like others in the thread but didn’t really get that far.

It did successfully find the C/C++ bugs I was looking for once the queries had hours of investment from both sides, and we were also able to find bugs in a large custom JS codebase by mocking all the things it would need to understand to eval the code. Whether or not that investment makes sense is an individual/team/org question, but if they ever seriously invest in a query debugger and profiling, they’ll be pretty hard to beat.

Re: Welcoming Semmle to GitHub

#95
post #47

Earlier quoted context omitted.

The Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?

It looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revisio…

Right. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)

Re: Welcoming Semmle to GitHub

#96

Earlier quoted context omitted.

"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later af…

So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback. If your data is closer to a model where security bug hunters whos…

[deleted]

Re: Welcoming Semmle to GitHub

#97
post #55

Earlier quoted context omitted.

Who have you talked to about it? Outside Mozilla and Microsoft?

I’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far bett…

Fair! Thanks!

Re: Welcoming Semmle to GitHub

#98
post #76

Earlier quoted context omitted.

I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.

Anecdata, I have also successfully used it to unearth complicated bugs that I already knew must exist, but didn’t have a good way to find. Think of a query like: Find all calls to function A that have an output pointer-pointer of type B in the last argument position and also have a boolean return type, verify the input dereferenced B is NULL, then verify that iff A assigns output to B that the return type is false an…

This is exactly the kind of thing I'm interested in knowing about Semmle. Thank you!

Re: Welcoming Semmle to GitHub

#99

Earlier quoted context omitted.

Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams…

"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later af…

Fermin here. Danny, I loved working with you at Google but let me correct you about my promotion and money. Not true.

I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :)

And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this.

Happy to sync in private over a coffee!

Re: Welcoming Semmle to GitHub

#100

Earlier quoted context omitted.

Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams…

"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later af…

Danny, I am the CEO and founder of Semmle. I will refrain from arguing about the value of our product and technology. However, I must correct your statement about Fermín which is utterly false. He took a huge pay cut to come to Semmle. Please stick to facts when talking about people.
Post reply on HN