Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Glean – System for collecting, deriving and querying facts about source code

glean.software

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Glean – System for collecting, deriving and querying facts about source code

#31

Post by simonmar »

coderdd wrote: post #21

Great to see this space moving! Any pointers on diff vs Kythe? Kythe has a mostly fixed schema, for one. One of the pain points using Kythe is wiring up the indexer to the build system. Would Glean indexers be easier to wire up for the common cases? Other is the index post-processing, which is not very scalable in the open source version (due to go-beam having rough Flunk support, for example). Third, how does it lin…

Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/s...

For wiring up the indexer, there are various methods, it tends to depend very much on the language and the build system. For Flow for example, Glean output is just built into the typechecker, you just run it with some flags to spit out the Glean data. For C++, you need to get the compiler flags from the build system to pass to the Clang frontend. For Java the indexer is a compiler plugin; for Python it's built on libCST. Some indexers send their data directly to a Glean server, others generate files of JSON that get sent using a separate command-line tool.

References use different methods depending on the language. For Flow for example there is a fact for an import that matches up with a fact for the export in the other file. For C++ there are facts that connect declarations with definitions, and references with declarations.

Re: Glean – System for collecting, deriving and querying facts about source code

#32

Post by scns »

lazamar wrote: post #7
soonnow wrote: post #4

I had a look at the site and it seems to be parsing source code in multiple languages and storing the parsed "syntax trees" into a database for querying. I would love to know what the usecase for this tool is aside from maybe being a source for presentations? (We have 5 million if statements). How can this be used to improve code quality or any other aspect of the code lifecycle? Or is it solving problems in a comple…

Glean is focused on storing and querying data about the code. The idea is that you have your own program to collect that data, then you use Glean to store that compactly and to have snappy queries. You would create entries like "this is a declaration of X", "this is a use of X". Then you can query things like "give me all uses of X" in sub-millisecond time. You hook that up to an LSP server then you get almost zero-c…

Oh wow, mindblowing stuff. Glad to see tech like this being open sourced, fuels the imagination about possible future scenarios. Do you use it on the Linux Kernel?

Re: Glean – System for collecting, deriving and querying facts about source code

#33

Post by dmos62 »

progval wrote: post #17
dons wrote: post #5

We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…

How would it perform for, say, 500TB of source code? And what would be the disk and memory requirements for this? Could they be distributed across a handful of servers?

I'd be surprised if this question could have an off hand answer. Doesn't sound like something that could have scalability predictable enough to do back of the envelope calculations on.

Re: Glean – System for collecting, deriving and querying facts about source code

#34

Post by Game_Ender »

simonmar wrote: post #31
coderdd wrote: post #21

Great to see this space moving! Any pointers on diff vs Kythe? Kythe has a mostly fixed schema, for one. One of the pain points using Kythe is wiring up the indexer to the build system. Would Glean indexers be easier to wire up for the common cases? Other is the index post-processing, which is not very scalable in the open source version (due to go-beam having rough Flunk support, for example). Third, how does it lin…

Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/s... For wiring up the indexer, there are various methods, it tends to dep…

Is there an example of using the C++ indexer? I saw hack and JS on your site but missed C++ (Python would also be amazing!).

Re: Glean – System for collecting, deriving and querying facts about source code

#35

Post by scns »

muglug wrote: post #29
erlich wrote: post #23

I can't believe Facebook hasn't canned Flowtype yet and moved to TypeScript. They will have to do it eventually.

I'm not sure you understand the scale at which Facebook operates. They don't have to do anything. As long as billions of people keep using Facebook they can maintain their own static analysis tooling for Javascript for as long as they want.

You do have a point, a rewrite on that scale would be a colossal waste of manyears/$$. Your delivery could be nicer though.

Re: Glean – System for collecting, deriving and querying facts about source code

#36

Post by minxomat »

scns wrote: post #32
lazamar wrote: post #7

Earlier quoted context omitted.

Glean is focused on storing and querying data about the code. The idea is that you have your own program to collect that data, then you use Glean to store that compactly and to have snappy queries. You would create entries like "this is a declaration of X", "this is a use of X". Then you can query things like "give me all uses of X" in sub-millisecond time. You hook that up to an LSP server then you get almost zero-c…

Oh wow, mindblowing stuff. Glad to see tech like this being open sourced, fuels the imagination about possible future scenarios. Do you use it on the Linux Kernel?

A comparable, powerful system (CodeQL) was used recently on the kernel[1] and Chrome. You can learn more about it here: https://codeql.github.com/docs/codeql-overview/about-codeql/

(disclosure: I work at GH on CQL)

[1] https://pwning.systems/posts/sequoia-variant-analysis/

Re: Glean – System for collecting, deriving and querying facts about source code

#40

Post by simonmar »

Game_Ender wrote: post #34
simonmar wrote: post #31

Earlier quoted context omitted.

Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/s... For wiring up the indexer, there are various methods, it tends to dep…

Is there an example of using the C++ indexer? I saw hack and JS on your site but missed C++ (Python would also be amazing!).

We want to open-source the C++ and Python indexers but they're not ready yet - we have to separate them from internal build-system-specific bits.
Post reply on HN