Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

51–60 of 164 posts

Re: Code search is hard

#51

Post by bawolff »

isker wrote: post #32
bawolff wrote: post #17

Surprised that hound https://github.com/hound-search/hound isn't mentioned. I thought it was the leader of open source solutions in this space. I've been using Wikimedia's instance ( https://codesearch.wmcloud.org/search/ ) and have generally been pretty happy with what it provides.

Hound has made an interesting choice to not bound searches. https://codesearch.wmcloud.org/search/?q=test&files=&exclude... produces an ajax request that (for me) took 13s to produce a 55MB JSON response, and then takes many more seconds to render into the DOM. Properly bounding search response sizes was one of the things I had to ensure Zoekt could do in its JSON APIs that I use in neogrok: https://github.com/source…

Yeah, i agree, that is weird. Especially if you search for something super common like "function" you basically DoS it.

Re: Code search is hard

#52

Post by bevekspldnw »

sdesol wrote: post #25

> It’s hard to find any accounts of code-search using FTS I'm actually going to be doing this soon. I've thought about code search for close to a decade, but I walked away from it, because there really isn't a business for it. However, now with AI, I'm more interested in using it to help find relevant context and I have no reason to believe FTS won't work. In the past I used Lucene, but I'm planning on going all in w…

Mmm, it’s not that straight forward: indexes can vastly slow down large scale ingest, so it’s really about when to index as well.

I work with a lot of multi billion row datasets and a lot of my recent focus has been on developing strategies to avoid the slow down with ingest, and then enjoying the speed up for indexed on search.

I’ve also gotten some mjnd boggling speed increases by summarizing key searchable data in smaller tables, some with JSONB columns that are abstractions of other data, indexing those, and using pg prewarm to serve those tables purely from memory. I literally went from queries taking actual days to < 1 sec.

Re: Code search is hard

#53

Post by worldsayshi »

samatman wrote: post #34
worldsayshi wrote: post #10

I suppose using something like tree sitter to get a consistent abstract syntax tree to work with would be a good starting point. And then try building a custom analyzer (if using elasticsearch lingo) with that?

Might be overkill unless you're looking to do semantic search. I've thought about what a search DSL for code would look like, it's challenging to embody a query like "method which takes an Int64 and has a variable idx in it" into something compact and memorable. But a tokenizer seems like a good place to start, I think that's the right granularity for this kind of application. You'd want to do some chunking so that f…

Sure you should definitively not try to do the overkill use case first but I would assume that tree sitter can emit "just" tokens as well? Getting the flexibility and control of a tool like tree sitter should allow you to quickly throw away stuff like comments and keywords if you want since you can do syntax aware filtering.

Then again I haven't used tree-sitter, can just imagine that this is a strength of it.

Re: Code search is hard

#54

Post by mdaniel »

scubbo wrote: post #49
keybored wrote: post #38

Earlier quoted context omitted.

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy

But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?

Re: Code search is hard

#55

Post by boyter »

Code search is indeed hard. Stop words, stemming and such do rule out most off the shelf indexing solutions but you can usually turn them off. You can even get around the splitting issues of things like

    a.toString()
With some pre-processing of the content. However were you really get into a world of pain is allowing someone to search for ring in the example. You can use partial term search, prefix, infix, or suffix but this massively bloats the index and is slow to run.

The next thing you try is trigrams, and suddenly you have to deal with false positive matches. So you add a positional portion to your index, and all of a sudden the underlying index is larger than the content you are indexing.

Its good fun though. For those curious about it I would also suggest reading posts by Michael Stapelberg https://michael.stapelberg.ch/posts/ who writes about Debian Code Search (which I believe he started) in addition to the other posts mentioned here. Shameless plug, I also write about this https://boyter.org/posts/how-i-built-my-own-index-for-search... where I go into some of the issues when building a custom index for searchcode.com

Oddly enough I think you can go a long way brute forcing the search if you don't do anything obviously wrong. For situations where you are only allowed to search a small portion of the content, say just your own (which looks applicable in this situation) that's what I would do. Adding an index is really only useful when you start searching at scale or you are getting semantic search out of it. For keywords which is what the article appears to be talking about, that's what I would be inclined to do.

Re: Code search is hard

#56

Post by worldsayshi »

morgante wrote: post #50
sqs wrote: post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

I've been surprised at how far you can get without indexing. Ex. I always assume we'll need to add an index to speed up GritQL ( https://github.com/getgrit/gritql ), but we've gotten pretty far with doing search entirely on the fly.

What does 'on the fly' entail here?

Re: Code search is hard

#57

Post by simonw »

worldsayshi wrote: post #56
morgante wrote: post #50

Earlier quoted context omitted.

I've been surprised at how far you can get without indexing. Ex. I always assume we'll need to add an index to speed up GritQL ( https://github.com/getgrit/gritql ), but we've gotten pretty far with doing search entirely on the fly.

What does 'on the fly' entail here?

I'm going to guess brute force - scan everything for the search term, rather than trying to use an index.

I'm always amazed at how fast ripgrep (rg) can brute force it's way through hundreds of MBs of source code.

Re: Code search is hard

#58

Post by MH15 »

healeycodes wrote: post #3

When a val is deployed on val town, my understanding is that it's parsed/compiled. At that point, can you save the parts of the program that people might search for? Names of imports, functions, variables, comments, etc.

A val is just Typescript, no? So unless they are also storing the AST it would be JavaScript and that's it

Re: Code search is hard

#59

Post by anonymousDan »

CityOfThrowaway wrote: post #23
jackbravo wrote: post #13

Would LLM vector embeddings work in this context? I'm guessing they should since they are very good at understanding code.

Yes but generating that index would be expensive

Why exactly? You mean to construct the embeddings or to embed the queries?
Post reply on HN