Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

141–150 of 164 posts

Re: Code search is hard

#141

Post by kermatt »

Are any of the tools mentioned in these comments better suited to searching SQL code, both DML and DDL?

We maintain a tree of files with each object in a separate "CREATE TABLE|VIEW|PROCEDURE|FUNCTION" script. This supports code search with grep, but something that could find references to an object when the name qualifications are not uniform would be very useful:

INSERT INTO table INSERT INTO schema.table INSERT INTO database.schema.table

Can all be done with regex, but search is not so easy for programmers new to expressions.

Re: Code search is hard

#142

Post by omitmyname »

isker wrote: post #19
peter_l_downs wrote: post #6

Surprised not to see Livegrep [0] on the list of options. Very well-engineered technology; the codebase is clean (if a little underdocumented on the architecture side) and you should be able to index your code without much difficulty. Built with Bazel (~meh, but useful if you don't have an existing cpp toolchain all set up) and there are prebuilt containers you can run. Try that first. By the way, there's a demo runn…

When I investigated using livegrep for code search at work, it really struggled to scale to a large number of repositories. At least at the time (a few years ago) indexing in livegrep was a monolithic operation: you index all repos at once, which produces one giant index. This does not work well once you're past a certain threshold. I also recall that the indexes it produces are pretty heavyweight in terms of memory…

Oh my god. This is amazing. I was thinking of building such thing myself. Thank you!

Re: Code search is hard

#143

Post by FalconSensei »

sqs wrote: post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

Do you plan on ever allowing users to change the font size?

Re: Code search is hard

#144

Post by philippemnoel »

fizx wrote: post #80

There's a million paths, but here's one I like. Use ElasticSearch. It will scale more than Postgres. Three hosted options are AWS, Elastic, Bonsai. I founded Bonsai and retired (so am partial), but they will provide the best human support for you, and you won't have to worry about java Xmx. Your goal with ES is to use the Regex PatternAnalyzer to split the code into reasonable exact code-shaped tokens (not english wo…

Elasticsearch is good, and it does scale, but it is much more cumbersome and expensive to scale and operate than Postgres. If you use the managed service, you'll pay for the operational pain in the form of higher pricing.

The Postgres movement is strong and extensions like ParadeDB https://github.com/paradedb/paradedb are designed specifically to solve this pain point (Disclaimer: I work for ParadeDB)

Re: Code search is hard

#145

Post by philippemnoel »

nojvek wrote: post #128

You can do to_tsvector “plain” and keep the strings intact. No lemming, stemming. We use plain tsvectors on a gin index and change the queries to allow prefix based searching. So “wo he” matches “hello world”. Perhaps I should write a blog about it. Took me a few days to read PG documentation to get where we are at. The only thing it doesn’t handle is typo tolerance.

Tsvector is amazing and it goes a long way, but unfortunately as you say it lacks some of more complex FTS features like typo tolerance, language tokenizers, etc.

Re: Code search is hard

#146

Post by philippemnoel »

bevekspldnw wrote: post #52
sdesol wrote: post #25

> It’s hard to find any accounts of code-search using FTS I'm actually going to be doing this soon. I've thought about code search for close to a decade, but I walked away from it, because there really isn't a business for it. However, now with AI, I'm more interested in using it to help find relevant context and I have no reason to believe FTS won't work. In the past I used Lucene, but I'm planning on going all in w…

Mmm, it’s not that straight forward: indexes can vastly slow down large scale ingest, so it’s really about when to index as well. I work with a lot of multi billion row datasets and a lot of my recent focus has been on developing strategies to avoid the slow down with ingest, and then enjoying the speed up for indexed on search. I’ve also gotten some mjnd boggling speed increases by summarizing key searchable data in…

That's wild. Quite impressive how far Postgres can be tuned. Is this all with tsvector?

Re: Code search is hard

#147

Post by philippemnoel »

johnthescott wrote: post #87

the rum index has worked well for us on roughly 1TB of pdfs. written by postgrespro, same folks who wrote core text search and json indexing. not sure why rum not in core. we have no problems. https://github.com/postgrespro/rum

RUM is good, but it lacks some of the more complex features like language tokenizers, etc. that a full search engine library like Lucene/Tantivy (and ParadeDB in Postgres) offer

Re: Code search is hard

#148

Post by omitmyname »

omitmyname wrote: post #142
isker wrote: post #19

Earlier quoted context omitted.

When I investigated using livegrep for code search at work, it really struggled to scale to a large number of repositories. At least at the time (a few years ago) indexing in livegrep was a monolithic operation: you index all repos at once, which produces one giant index. This does not work well once you're past a certain threshold. I also recall that the indexes it produces are pretty heavyweight in terms of memory…

Oh my god. This is amazing. I was thinking of building such thing myself. Thank you!

is there any way to open file like in zoekt? it's so much better than native zoekt ui except this:(

Re: Code search is hard

#149

Post by ivanovm »

One of the most interesting approaches to code search I've seen recently (no affiliation) https://github.com/pyjarrett/septum

The hardest part about getting code search right imo is grabbing the right amount of surrounding context, which septum is aimed at solving on a per-file basis.

Another one I'm surprised hasn't been mentioned is stack-graphs (https://github.com/github/stack-graphs), which tries to incrementally resolve symbolic relationships across the whole codebase. It powers github's cross-file precise indexing and conceptually makes a lot of sense, though I've struggled to get the open source version to work

Re: Code search is hard

#150

Post by ivanovm »

jackbravo wrote: post #13

Would LLM vector embeddings work in this context? I'm guessing they should since they are very good at understanding code.

I've found embeddings to perform quite poorly on code because 1) user queries are not semantically similar to target code in most cases 2) often times two very concretely related pieces of code are not at all semantically similar
Post reply on HN