I would use Hound.
Code search is hard
11–20 of 164 posts
Re: Code search is hard
#12any nuggets here? https://github.blog/2023-02-06-the-technology-behind-githubs...
Would also recommend the related video
https://www.youtube.com/watch?v=CqZA_KmygKw
Re: Code search is hard
#13Would LLM vector embeddings work in this context? I'm guessing they should since they are very good at understanding code.
Re: Code search is hard
#14It indeed is hard, and a good code search platform makes life so much easier. If I ever leave Google, the internal code search is for sure going to be the thing I miss the most. It's so well integrated into how everything else works (blaze target finding, guice bindings etc), I can't imagine my life without it. I remember to appreciate it even more every time I use Github's search. Not that it's bad, it's just inhere…
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs.
Re: Code search is hard
#15I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing to search everything exhaustively). Happy to chat with anyone who's building this kind of thing, including with folks at Val Town, which is awesome.
Re: Code search is hard
#16I suppose using something like tree sitter to get a consistent abstract syntax tree to work with would be a good starting point. And then try building a custom analyzer (if using elasticsearch lingo) with that?
Another option is to start with Kythe, which is Google's own open source framework for getting a uniform cross-language semantic layer: https://kythe.io/
Worth looking at as a source of inspiration and design ideas even if you don't want to use it itself.
Re: Code search is hard
#17Surprised that hound https://github.com/hound-search/hound isn't mentioned. I thought it was the leader of open source solutions in this space.
I've been using Wikimedia's instance ( https://codesearch.wmcloud.org/search/ ) and have generally been pretty happy with what it provides.
Re: Code search is hard
#18It indeed is hard, and a good code search platform makes life so much easier. If I ever leave Google, the internal code search is for sure going to be the thing I miss the most. It's so well integrated into how everything else works (blaze target finding, guice bindings etc), I can't imagine my life without it. I remember to appreciate it even more every time I use Github's search. Not that it's bad, it's just inhere…
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep
I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do.
Sourcegraph is the closest thing I know of.
Re: Code search is hard
#19Surprised not to see Livegrep [0] on the list of options. Very well-engineered technology; the codebase is clean (if a little underdocumented on the architecture side) and you should be able to index your code without much difficulty. Built with Bazel (~meh, but useful if you don't have an existing cpp toolchain all set up) and there are prebuilt containers you can run. Try that first. By the way, there's a demo runn…
When I investigated using livegrep for code search at work, it really struggled to scale to a large number of repositories. At least at the time (a few years ago) indexing in livegrep was a monolithic operation: you index all repos at once, which produces one giant index. This does not work well once you're past a certain threshold.
I also recall that the indexes it produces are pretty heavyweight in terms of memory requirements, but I don't have any numbers on hand to justify that claim.
Zoekt (also mentioned in TFA) has the correct properties in this regard. Except in niche configurations that are probably only employed at sourcegraph, each repo is (re)indexed independently and produces a separate set of index files.
But its builtin web UI left much to be desired (especially compared to livegrep), so I built one: https://github.com/isker/neogrok.
Re: Code search is hard
#20Earlier quoted context omitted.
If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep
I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.
Agreed. There are some public building blocks available (e.g. Kythe or meta's Glean) but having something generic that produces the kind of experience you can get on cs.chromium.org seems impossible. You need such bespoke build integration across an entire organization to get there.
Basic text search, as opposed to navigation, is all you'll get from anything out of the box.