This seems very interesting, would love to see more alternatives to TreeSitter and microsoft LSP - what makes those hard to use is lack of examples and tutorials. So I hope tbere will be examples and tutorials. For example: How do you find all variables in scope when the text cursor is on line x and col y in /file/path/file.js
Glean – System for collecting, deriving and querying facts about source code
51–60 of 85 posts
Re: Glean – System for collecting, deriving and querying facts about source code
#52I was recently looking for a library that takes a few lines of source code as input, and predicts the programming language as output. That seems like a very tractable machine learning problem, yet all I could find was a single python library which looks nice, but doesn't have much adoption, and requires installing the entirety of tensorflow despite the fact that users just want a trained model and a predict() functio…
GitHub's linguist library can be used to identify the programming language of a single file (edit: or of a whole project): https://github.com/github/linguist#single-file
Re: Glean – System for collecting, deriving and querying facts about source code
#53Datalog-ish query languages sure is a fun area to be working in. Such DSLs exist for various domains and, like Semmle's codeQL or the more academic Soufflé, Glean focuses on the domain of programming languages. Glean seems to still be work in progress, e.g. no support for recursive queries yet, but I wonder where they're heading. I'll certainly keep an eye on the project but I wonder how exactly Glean aims to -- or m…
I think the point in the space Glean hits well is efficiency/latency (enough to power real time editing, like in IDE autocomplete or navigation), while having a schema and query language generic enough to do multiple languages and code-like things. You can accurately query JavaScript or Rust or PHP or Python or C++ with a common interface, which is a bit nuts :D
Re: Glean – System for collecting, deriving and querying facts about source code
#54Earlier quoted context omitted.
How would it perform for, say, 500TB of source code? And what would be the disk and memory requirements for this? Could they be distributed across a handful of servers?
What on earth has this much source code? Every open source project ever?
Yes, good guess! That's the size we have after deduplication across projects at https://www.softwareheritage.org/ . We archive all the source code we can find; and would like to support some sort of full-text search on it at some point, so Glean looks interesting
Re: Glean – System for collecting, deriving and querying facts about source code
#557GB docker image !
Re: Glean – System for collecting, deriving and querying facts about source code
#56But whats an example of a fact? Looks cool but I have no idea what its for.
Re: Glean – System for collecting, deriving and querying facts about source code
#57I was recently looking for a library that takes a few lines of source code as input, and predicts the programming language as output. That seems like a very tractable machine learning problem, yet all I could find was a single python library which looks nice, but doesn't have much adoption, and requires installing the entirety of tensorflow despite the fact that users just want a trained model and a predict() functio…
GitHub's linguist library can be used to identify the programming language of a single file (edit: or of a whole project): https://github.com/github/linguist#single-file
Thanks! My searches completely failed to find that. I can’t use it as a ruby library, but perhaps I can pull out the heuristics.yml and the naive bayes classifier weights to use in another language.
Re: Glean – System for collecting, deriving and querying facts about source code
#58We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…
I'm really struggling to understand what Glean does, and why I would use it. Most important: Your landing page should quickly show what Glean does that a typical IDE (Visual Studio, Visual Studio Code, Eclipse, ect, does.) Specifically, things like "Go to definition," and tab completion have been in industry-leading IDEs for at least 20 years. What's novel about Glean? It seems like a lot of hoops to jump through whe…
> It seems like a lot of hoops to jump through when Visual Studio (and Visual Studio Code) can index a very large codebase in a few seconds.
I think you are not thinking large enough. An IDE absolutely can not index a very large codebase and allow users to make complex queries on it. Think multiple millions lines of code here. The use case is closer to "find me all the variables of this type or a type derived from it in all the projects at Facebook" than "go to this definition in the project I'm currently editing".
Re: Glean – System for collecting, deriving and querying facts about source code
#59To prevent any confusion, this is a different product than Mozilla's Glean [0][1].
[0] https://docs.telemetry.mozilla.org/concepts/glean/glean.html
Re: Glean – System for collecting, deriving and querying facts about source code
#60We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros…
I'm really struggling to understand what Glean does, and why I would use it. Most important: Your landing page should quickly show what Glean does that a typical IDE (Visual Studio, Visual Studio Code, Eclipse, ect, does.) Specifically, things like "Go to definition," and tab completion have been in industry-leading IDEs for at least 20 years. What's novel about Glean? It seems like a lot of hoops to jump through whe…
What size codebases do you have that a few seconds has visual studio fully indexing it? My experience with VS on large projects is that it takes however long the project takes to compile before it's usable, but many functions (go to definition) can occasionally hit a file that needs to be reparsed and can stall for minutes on end. I use Vs2019 on a 32 core workstation with 128GB ram, fwiw.