Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

tzx.notion.site

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#41

Post by zxt_tzx »

whakim wrote: post #37

I was the first employee at a company which uses RAG (Halcyon), and I’ve been working through issues with various vector store providers for almost two years now. We’ve gone from tens of thousands to billions of embeddings in that timeframe - so I feel qualified to at least offer my opinion on the problem. I agree that starting with pgvector is wise. It’s the thing you already have (postgres), and it works pretty wel…

Thank you for the comment, compared to you I have only touched the bare surface of this quite complex domain, would love to get more of your input!

> building HNSW indices in Postgres is still extremely slow (even with parallel index builds), so it is difficult to experiment with index hyperparameters at scale.

Yes, I experienced this too. I from 1536 to 256 and did not try more values than I'd have liked because spinning up a new database and recreating the embeddings simply took too long. I’m glad it worked well enough for me, but without a quick way to experiment with these hyperparameters, who knows whether I’ve struck the tradeoff at the right place.

Someone on Twitter reached out and pointed out one could quantizing the embeddings to bit vectors and search with hamming distance — supposedly the performance hit is actually very negligible, especially if you add a quick rescore step: https://huggingface.co/blog/embedding-quantization

> But (as mentioned in other comments) keeping your data in sync is a huge issue.

Curious if you have any good solutions in this respect.

> The other challenge I’ve found is that filtering is often the “special sauce” that vector store providers bring to the table, so it’s pretty difficult to reason about the performance and recall of various types of filters.

I realize they market heavily on this, but for open source databases, wouldn't the fact that you can see the source code make it easier to reason about this? or is your point that their implementation here are all custom and require much more specialized knowledge to evaluate effectively?

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#42

Post by snikolaev »

zxt_tzx wrote: post #28
nchmy wrote: post #18

This seems pretty similar to something that the ManticoreSearch team released a year ago https://manticoresearch.com/blog/manticoresearch-github-issu... You can index any GH repo and then search it with vector, keyword, hybrid and more. There's faceting and anything else you could ever want. And it is astoundingly fast - even vector search. Here's the direct link to the demo https://github.manticoresearch.com/

oh wow that's super cool, I tried it and it's very fast indeed. thanks for sharing! will spend more time to understand how it's implemented

This article may be more relevant https://manticoresearch.com/blog/github-semantic-search/

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#43

Post by franky47 »

zxt_tzx wrote: post #32
franky47 wrote: post #12

I started a quick weekend project to do just that today: index my OSS project's [1] issues & discussions, so I can RAG-ask it to find references when I feel like I'm repeating myself (in "see issue/PR/discussion #123", finding the 123 is the hardest part). This article might be super helpful, thanks! I don't intend to make a product out of it though, so I can cut a lot of corners, like using a PAT for auth and runnin…

After this failed experience with SemHub, I am actually thinking of building something like this, for open source maintainers like you are definitely the ICP! (nuqs seems really cool btw, storing state in the URL param is definitely the way to go) To elaborate, I was thinking of: - running a cron that checks repos every X minutes - for every new issue someone has opened, I will run an agent that (1) checks e.g. SemHu…

While having a bot that auto-replies with "similar issues" pointers might make sense at a large scale (to relieve maintainers), I usually prefer to do this manually at my current scale, knowing that there's one particular instance where I pointed someone in a given direction, and want to either reuse/modify a code example block, or stitch together semantically unrelated but relevant comments & discussions together.

You might want to talk to Jovi [1] about that, he's doing something very similar.

[1] https://bsky.app/profile/jovidecroock.com/post/3lh6hkcxnqc2v

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#44

Post by ichiwells »

whakim wrote: post #37

I was the first employee at a company which uses RAG (Halcyon), and I’ve been working through issues with various vector store providers for almost two years now. We’ve gone from tens of thousands to billions of embeddings in that timeframe - so I feel qualified to at least offer my opinion on the problem. I agree that starting with pgvector is wise. It’s the thing you already have (postgres), and it works pretty wel…

> Finally, building HNSW indices in Postgres is still extremely slow (even with parallel index builds), so it is difficult to experiment with index hyperparameters at scale

For anyone coming across this without much experience here, for building these indexes in pgvector it makes a massive difference to increase your maintenance memory above the default. Either as a separate db like whakim mentioned, or for specific maintenance periods depending on your use case.

``` SHOW maintenance_work_mem; SET maintenance_work_mem = X; ```

In one of our semantic search use cases, we control the ingestion of the searchable content (laws, basically) so we can control when and how we choose to index it. And then I've set up classic relational db indexing (in addition to vector indexing) for our quite predictable query patterns.

For us that means our actual semantic db query takes about 10ms.

Starting from 10s of millions of entries, filtered to ~50k (jurisdictionally, in our case) relevant ones and then performing vector similarity search with topK/limit.

Built into our ORM and zero round-trip latency to Pinecone or syncing issues.

EDIT: I imagine whakim has more experience than me and YMMV, just sharing lesson learned. Even with higher maintenance mem the index building is super slow for HNSW

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#45

Post by Noumenon72 »

johnfn wrote: post #9

That was a great write up. If you don't mind me giving you some unsolicited product feedback: I think SemHub didn't do well because it's unclear what problem it's actually solving. Who actually wants your product? What's the use case? I use GitHub issues all the time, and I can't think of a reason I'd want semhub. If I need to find a particular issue on, say, TypeScript, I'll just google "github typescript issue [des…

https://manticoresearch.com/blog/github-semantic-search/ gives some good examples where you get more with semantic than keyword search:

  * Search for "memory leak", get "index out of memory"
  * Search "API rate limits", get “throttling”, “250 results” limit, and “rate limiting”
  * Search issues for "user authentication" to see whether anyone has submitted your feature request
  * Search for “SQL injection” to get “database infiltration” or “SQL vulnerability”

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#46

Post by ljm »

kevmo314 wrote: post #11

It's somewhat ironic that the author advocates for keeping it simple and using pgvector but then buries a ton of complexity with an API server, auth server, Cloudflare workers, and durable objects. Especially given > Supabase easily the most expensive part of my stack (at $200/month, if we ran in it XL, i.e. the lowest tier with 4-core CPU) That could get you a pretty decent VPS and allow you to coassemble everything…

Not speaking for OP’s experience but I suppose that you might default to all this fancy serverless edge worker stuff if you learned how to code on their (usually generous) free-tier plans, or they were the only things you dealt with at work.

Meanwhile setting up a little VPS box would come more naturally if you learned in the era of the LAMP stack and got your hands dirty with Linux.

In fact I wonder if for some people that’s made worse by the tendency to split frontend and backend web development into completely separate disciplines when originally you did the whole thing.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#47

Post by johnfn »

Noumenon72 wrote: post #45
johnfn wrote: post #9

That was a great write up. If you don't mind me giving you some unsolicited product feedback: I think SemHub didn't do well because it's unclear what problem it's actually solving. Who actually wants your product? What's the use case? I use GitHub issues all the time, and I can't think of a reason I'd want semhub. If I need to find a particular issue on, say, TypeScript, I'll just google "github typescript issue [des…

https://manticoresearch.com/blog/github-semantic-search/ gives some good examples where you get more with semantic than keyword search: * Search for "memory leak", get "index out of memory" * Search "API rate limits", get “throttling”, “250 results” limit, and “rate limiting” * Search issues for "user authentication" to see whether anyone has submitted your feature request * Search for “SQL injection” to get “databas…

I understand that semantic search gives you more than keyword search. I don't understand the cases when you need that. Like I said, I use GitHub issues all the time, and it's exceedingly rare that I need to search for something extensively like this.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#48

Post by johnfn »

zxt_tzx wrote: post #27
johnfn wrote: post #9

That was a great write up. If you don't mind me giving you some unsolicited product feedback: I think SemHub didn't do well because it's unclear what problem it's actually solving. Who actually wants your product? What's the use case? I use GitHub issues all the time, and I can't think of a reason I'd want semhub. If I need to find a particular issue on, say, TypeScript, I'll just google "github typescript issue [des…

Thanks for the feedback, to be honest, my own experience is actually very similar to yours. The original pain point probably only exists for small minority of open source maintainers who manage multiple repos and actually search across them regularly. Most devs are probably like you and I, and the mediocre GitHub search experience is more than compensated by using Google. In its current iteration, it's quite hard to…

Any chance you live in SF? If so, we should meet up - I'm working on something similar. You can reach out at my username @gmail.com

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#49

Post by VirgilShelton »

Hey Warren great job on the site, but what you'll need to do is SEO. You're a great writer so all you need to add to your writing skills is SEO. I did a basic SEO audit of semhub.dev and you have no SEO. While this is niche you'll need to add a blog to your website and use basic SEO keyword research to find what your target audience is searching for instead of just blogging to blog. Start reading https://backlinko.com/seo-basics-for-beginners and you'll be well on your way. It should take about a year for you to get some good traction. Don't rush, just keep learning more and more everyday and you'll get there in a few years with organic SEO alone. The comments here alone are proof that you have a viable MVP.

GL!

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#50

Post by dgellow »

scottyeager wrote: post #36

> * No way to search across multiple repos within GitHub. > * No way to easily see open and closed issues in the same view. I don't quite understand, because searching issues across all of Github and also within orgs is already supported. Those searches show both open and closed issues by default. For searches on a single repo, just removing the "state" filter entirely from the query also shows open and closed issues…

GitHub search is pretty unreliable from my experience. Search results are limited to 1000 items, and you never know if the index you’re searching against is up to date — unless a file has been opened recently in GitHub web UI there is a significant and unpredictable delay between a commit and the indexing.

So far I’ve been very happy with Livegrep, we are using it to search across ~10k repos, the index is rebuilt once an hour with a simple cron job. Searching is insanely fast, it’s using very little resources, just a simple computer engine instance. The main downside is the lack of multiline search, but so far that hasn’t been too much of a problem.

Post reply on HN