Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

21–30 of 164 posts

Re: Code search is hard

#21

Post by tayo42 »

init wrote: post #18
peter_l_downs wrote: post #14

Earlier quoted context omitted.

If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep

I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.

Is there like a summary of what's missing from public attempts and what makes it so much better?

Re: Code search is hard

#22

Post by init »

isker wrote: post #20
init wrote: post #18

Earlier quoted context omitted.

I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.

Agreed. There are some public building blocks available (e.g. Kythe or meta's Glean) but having something generic that produces the kind of experience you can get on cs.chromium.org seems impossible. You need such bespoke build integration across an entire organization to get there. Basic text search, as opposed to navigation, is all you'll get from anything out of the box.

In a past job I built a code search clone on top of Kythe, Zoekt and LSP (for languages that didn't have bazel integration). I got help from another colleague to make the UI based on Monaco. We create a demo that many people loved but we didn't productionize it for a few reasons (it was an unfunded hackathon project and the company was considering another solution when they already had Livegrep)

Producing the Kythe graph from the bazel artifacts was the most expensive part.

Working with Kythe is also not easy as there is no documentation on how to run it at scale.

Re: Code search is hard

#24

Post by isker »

init wrote: post #22
isker wrote: post #20

Earlier quoted context omitted.

Agreed. There are some public building blocks available (e.g. Kythe or meta's Glean) but having something generic that produces the kind of experience you can get on cs.chromium.org seems impossible. You need such bespoke build integration across an entire organization to get there. Basic text search, as opposed to navigation, is all you'll get from anything out of the box.

In a past job I built a code search clone on top of Kythe, Zoekt and LSP (for languages that didn't have bazel integration). I got help from another colleague to make the UI based on Monaco. We create a demo that many people loved but we didn't productionize it for a few reasons (it was an unfunded hackathon project and the company was considering another solution when they already had Livegrep) Producing the Kythe g…

Very cool. I tried to do things with Kythe at $JOB in the past, but gave up because the build (really, the many many independent builds) precluded any really useful integration.

I did end up making a nice UI for vanilla Zoekt, as I mentioned elsewhere: https://github.com/isker/neogrok.

Re: Code search is hard

#25

Post by sdesol »

> It’s hard to find any accounts of code-search using FTS

I'm actually going to be doing this soon. I've thought about code search for close to a decade, but I walked away from it, because there really isn't a business for it. However, now with AI, I'm more interested in using it to help find relevant context and I have no reason to believe FTS won't work. In the past I used Lucene, but I'm planning on going all in with Postgres.

The magic to fast code search (search in general), is keeping things small. As long as your search solution is context aware, you can easily leverage Postgres sharding to reduce index sizes. I'm a strong believer in "disk space is cheap, time isn't", which means I'm not afraid to create as many indexes as required, to shave 100's of milliseconds of searches.

Re: Code search is hard

#26

Post by isker »

sqs wrote: post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

And when you're ready to do indexed search, Zoekt (over which Sourcegraph graciously took maintainership a while ago) is the best way to do it that I've found. After discounting both Livegrep and Hound (they both struggled to perform in various dimensions with the amount of stuff we wanted indexed, Hound moreso than Livegrep), we migrated to Zoekt from a (necessarily) very old and creaky deployment of OpenGrok and it's night and day, both in terms of indexing performance and search performance/ergonomics.

Sourcegraph of course adds many more sophisticated features on top of just the code search that Zoekt provides.

Re: Code search is hard

#27

Post by peter_l_downs »

isker wrote: post #19
peter_l_downs wrote: post #6

Surprised not to see Livegrep [0] on the list of options. Very well-engineered technology; the codebase is clean (if a little underdocumented on the architecture side) and you should be able to index your code without much difficulty. Built with Bazel (~meh, but useful if you don't have an existing cpp toolchain all set up) and there are prebuilt containers you can run. Try that first. By the way, there's a demo runn…

When I investigated using livegrep for code search at work, it really struggled to scale to a large number of repositories. At least at the time (a few years ago) indexing in livegrep was a monolithic operation: you index all repos at once, which produces one giant index. This does not work well once you're past a certain threshold. I also recall that the indexes it produces are pretty heavyweight in terms of memory…

I like this better than livegrep. I haven't actually operated either zoekt OR livegrep before, but I'll probably start with zoekt+neogrok next time I want to stand up a codesearch page. Thanks for building and sharing this!

Re: Code search is hard

#29

Post by Macha »

> This is a pretty bad index: it has words that should be stop words, like function, and won’t split a.toString() into two tokens because . is not a default word boundary.

So github used to (maybe still does) "fix" this one and it's annoying. Although github are ramping up their IDE like find-usages, it's still not perfect, so somethings you just want to a text search equivalent for "foo.bar()" for all the uses it misses and this stemming behaviour then finds every while where foo and bar are mentioned which bloats results.

Re: Code search is hard

#30

Post by sdesol »

tayo42 wrote: post #21
init wrote: post #18

Earlier quoted context omitted.

I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.

Is there like a summary of what's missing from public attempts and what makes it so much better?

The short answer is context. The reason why Google's internal code search is so good, is it is tied into their build system. This means, when you search, you know exactly what files to consider. Without context, you are making an educated guess, with regards to what files to consider.
Post reply on HN