If you're serious about scaling up, definitely consider Vespa (https://vespa.ai).
At serious scale, Vespa will likely knock all the other options out of the park.
Code search is hard
41–50 of 164 posts
Re: Code search is hard
#42Is it possible to combine n-gram and AST to dump a better indexing?
Take `sourceCode.toString()` as an example, the AST can dump it to `sourceCode` and `toString`. A further indexer can break `sourceCode` to `source` and `code`.
For ast dumping, project like https://github.com/ast-grep/ast-grep can help.
Re: Code search is hard
#43Earlier quoted context omitted.
The short answer is context. The reason why Google's internal code search is so good, is it is tied into their build system. This means, when you search, you know exactly what files to consider. Without context, you are making an educated guess, with regards to what files to consider.
How exactly integration with build system helps Google? Maybe you could give specific example?..
If you want to build a product with a build system, you need to tell it what source to include. With this information, you know what files to consider and if you are dealing with a statically typed language like C or C++, you have build artifacts that can tell you where the implementation was defined. All of this, takes the guess work out of answering questions like "What foo() implentation was used".
If all you know are repo branches, the best you can do is return matches from different repo branches with the hopes that one of them is right.
Edit: I should also add that with a build system, you know what version of a file to use.
Re: Code search is hard
#44> Sourcegraph’s maintained fork of Zoekt is pretty cool, but is pretty fearfully niche and would be a big, new infrastructure commitment.
I don't think Zoekt is as scary as this article makes it out to be. I set this up at my current company after getting experience with it at Shopify and its really great.
Re: Code search is hard
#45It indeed is hard, and a good code search platform makes life so much easier. If I ever leave Google, the internal code search is for sure going to be the thing I miss the most. It's so well integrated into how everything else works (blaze target finding, guice bindings etc), I can't imagine my life without it. I remember to appreciate it even more every time I use Github's search. Not that it's bad, it's just inhere…
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep
Just want to note that Livegrep, its antecedent "codesearch", and other things that are basically grep bear no resemblance to that which a person working at Google calls "Code Search".
Re: Code search is hard
#46Earlier quoted context omitted.
The short answer is context. The reason why Google's internal code search is so good, is it is tied into their build system. This means, when you search, you know exactly what files to consider. Without context, you are making an educated guess, with regards to what files to consider.
How exactly integration with build system helps Google? Maybe you could give specific example?..
Try clicking around https://source.chromium.org/chromium/chromium/src, which is built with Kythe (I believe, or perhaps it's using something internal to Google that Kythe is the open source version of).
By hooking into C++ compilation, Kythe is giving you things like _macro-aware_ navigation. Instead of trying to process raw source text off to the side, it's using the same data the compiler used to compile the code in the first place. So things like cross-references are "perfect", with no false positives in the results: Kythe knows the difference between two symbols in two different source files with the same name, whereas a search engine naively indexing source text, or even something with limited semantic knowledge like tree sitter, cannot perfectly make the distinction.
Re: Code search is hard
#47Earlier quoted context omitted.
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep
> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .
It's not just that. Livegrep isn't just a pale imitation of something inside Google. It's totally unrelated in implementation, capabilities, and use case.
Re: Code search is hard
#48Earlier quoted context omitted.
I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.
I see most replies here ar mentioning the the build integration is what is mainly missing in the public tools. I wonder if nix and nixpkgs could be used here? Nix is a language agnostic build-system and with nixpkgs it has a build instructions for a massive amount of packages. Artifacts for all packages are also available via hydra. Nix should also have enough context so that for any project it can get the source cod…
Build integration is not the main thing that is missing between Livegrep and Code Search. The main thing that is missing is the semantic index. Kythe knows the difference between this::fn(int) and this::fn(double) and that::fn(double) and so on. So you can find all the callers of the nullary constructor of some class, without false positives of the callers of the copy constructor or the move constructor. Livegrep simply doesn't have that ability at all. Livegrep is what it says it is on the box: grep.
Re: Code search is hard
#49Earlier quoted context omitted.
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep
> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .
I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close.
(I would be _delighted_ for someone to "Umm actually" me by providing a great product!)
Re: Code search is hard
#50I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…
I've been surprised at how far you can get without indexing.
Ex. I always assume we'll need to add an index to speed up GritQL (https://github.com/getgrit/gritql), but we've gotten pretty far with doing search entirely on the fly.