Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Glean – System for collecting, deriving and querying facts about source code

glean.software

Post reply on HN
Print view View original

41–50 of 85 posts

Re: Glean – System for collecting, deriving and querying facts about source code

#41

Post by Grimm1 »

Very cool! How does this differ algorithmically from the trigram based search that everything uses from google code search from like 20 years ago?

And continuing off of that theme in practical terms how does it stand up against zoekt?

I’m curious because zoekt is kind of slow when it comes to ingesting large amounts of code like all of the publicly available code on GitHub

The few people using that commercially have basically had to spend a lot of time rewriting parts of it to make their goal of public codesearch for all attainable.

I and a few people I know are pretty convinced that there are better and easier ways / technologies to make that happen.

Re: Glean – System for collecting, deriving and querying facts about source code

#42

Post by gricardo99 »

progval wrote: post #17
dons wrote: post #5

We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…

How would it perform for, say, 500TB of source code? And what would be the disk and memory requirements for this? Could they be distributed across a handful of servers?

What on earth has this much source code? Every open source project ever?

Re: Glean – System for collecting, deriving and querying facts about source code

#43

Post by gigatexal »

lazamar wrote: post #7
soonnow wrote: post #4

I had a look at the site and it seems to be parsing source code in multiple languages and storing the parsed "syntax trees" into a database for querying. I would love to know what the usecase for this tool is aside from maybe being a source for presentations? (We have 5 million if statements). How can this be used to improve code quality or any other aspect of the code lifecycle? Or is it solving problems in a comple…

Glean is focused on storing and querying data about the code. The idea is that you have your own program to collect that data, then you use Glean to store that compactly and to have snappy queries. You would create entries like "this is a declaration of X", "this is a use of X". Then you can query things like "give me all uses of X" in sub-millisecond time. You hook that up to an LSP server then you get almost zero-c…

Thank you for this summary I was unsure of how this is really useful. That before step is missing I think.

Re: Glean – System for collecting, deriving and querying facts about source code

#44

Post by gurleen_s »

gricardo99 wrote: post #42
progval wrote: post #17

Earlier quoted context omitted.

How would it perform for, say, 500TB of source code? And what would be the disk and memory requirements for this? Could they be distributed across a handful of servers?

What on earth has this much source code? Every open source project ever?

I mean, yeah. Imagine being able to do more rich queries against GitHub.

Re: Glean – System for collecting, deriving and querying facts about source code

#45

Post by zerr »

dons wrote: post #5

We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…

Since this is HN, could you please share more technical/impl details, e.g. what makes it more scalable and faster in general and also compared to other similar engines?

Re: Glean – System for collecting, deriving and querying facts about source code

#47

Post by wingspan »

erlich wrote: post #23

I can't believe Facebook hasn't canned Flowtype yet and moved to TypeScript. They will have to do it eventually.

The problem is that TypeScript does not scale to the size of the giant monorepo at Facebook, with hundreds of thousands, if not millions of files. Since they aren't organized into packages, it is just one giant flat namespace (any JS file can import any other JS file by the filename). It is pretty amazing to change a core file and see type errors across the entire codebase in a few seconds. The main way to scale in TypeScript is Project References, which don't work when you haven't separated your code into packages. (Worked at Facebook until June 2021).

Re: Glean – System for collecting, deriving and querying facts about source code

#48

Post by gwbas1c »

dons wrote: post #5

We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…

I'm really struggling to understand what Glean does, and why I would use it. Most important: Your landing page should quickly show what Glean does that a typical IDE (Visual Studio, Visual Studio Code, Eclipse, ect, does.)

Specifically, things like "Go to definition," and tab completion have been in industry-leading IDEs for at least 20 years.

What's novel about Glean? It seems like a lot of hoops to jump through when Visual Studio (and Visual Studio Code) can index a very large codebase in a few seconds. (And don't require a server and database to do it.)

Perhaps a 20-second video (no sound) showing what Glean does that other IDEs don't will help get the message across?

Re: Glean – System for collecting, deriving and querying facts about source code

#49

Post by jamessb »

simonw wrote: post #24

Feature request: a live demo! I would love to try out the web interface described at https://glean.software/docs/trying without pulling down a 7GB Docker image first.

Even just a short video of someone using the web interface would be helpful.

Re: Glean – System for collecting, deriving and querying facts about source code

#50

Post by balddenimhero »

Datalog-ish query languages sure is a fun area to be working in. Such DSLs exist for various domains and, like Semmle's codeQL or the more academic Soufflé, Glean focuses on the domain of programming languages.

Glean seems to still be work in progress, e.g. no support for recursive queries yet, but I wonder where they're heading. I'll certainly keep an eye on the project but I wonder how exactly Glean aims to -- or maybe it already does -- improve upon the alternatives? From the talk linked in another comment I guess the distinctive feature may be the planned integration with IDEs. Correct me if I'm wrong. Other contenders provide great querying technology but there is indeed no strong focus on making such tech really convenient and integrated yet.

Post reply on HN