Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

tzx.notion.site

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#21

Post by gfody »

zxt_tzx wrote: post #2

Author here. Over the last few months, I have built and launched a free semantic search tool for GitHub called SemHub ( https://semhub.dev/ ). In this blog post, I share what I’ve learned and why I’ve failed, so that other builders can learn from my experience. This blog post runs long and I have sign-posted each section. I have marked the sections that I consider the particularly insightful with an asterisk (*). I h…

it's weird you consider this a failure. you spent a few months and learned how to work with embedding models to build an efficient search. the fact that your search works well is a successful outcome. if your goal was to turn a few month effort into a thriving business that's never going to happen period - it only seems possible because when it does happen for people we completely discount the luck factor.

if you want to turn your search into a business now that's a new and different effort, mostly marketing and stuff that most self respecting engineers gives zero shits about, but if that's your real goal don't call it a failure yet because you haven't even tried.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#22

Post by romanhn »

zxt_tzx wrote: post #2

Author here. Over the last few months, I have built and launched a free semantic search tool for GitHub called SemHub ( https://semhub.dev/ ). In this blog post, I share what I’ve learned and why I’ve failed, so that other builders can learn from my experience. This blog post runs long and I have sign-posted each section. I have marked the sections that I consider the particularly insightful with an asterisk (*). I h…

Thanks for posting this, very timely as I'm also playing around with pgvector for semantic search. I saw that you ended up trimming inputs longer than 8K tokens. Have you looked into chunking (breaking input into smaller chunks and doing vector search on the chunks)? Embedding models I'm playing with have a max of 512 tokens, so chunking is pretty much a must. Choosing a chunking strategy seems to be a deep rabbit hole of its own.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#23

Post by zxt_tzx »

kevmo314 wrote: post #11

It's somewhat ironic that the author advocates for keeping it simple and using pgvector but then buries a ton of complexity with an API server, auth server, Cloudflare workers, and durable objects. Especially given > Supabase easily the most expensive part of my stack (at $200/month, if we ran in it XL, i.e. the lowest tier with 4-core CPU) That could get you a pretty decent VPS and allow you to coassemble everything…

Totally fair point. Thanks for taking the time to read through it! I guess I didn't want to use a VPS and then have to switch to something else if the product really worked, but I guess that rhymes with premature optimization.

Some other clarifications:

- I was also surprised with how expensive Supabase turned out to be and only got there because I was trying to sync very big repos ahead of time. I could see an alternative product where the cost here would be minimal too

- I did see this project as an opportunity to try out Cloudflare. as mentioned in the post, as a full stack TypeScript developer, I thought Cloudflare could be a good fit and I still really want it to succeed as a cloud platform

- deploying two separate API server and auth server is actually simpler than it sounds, since each is a Cloudflare Worker! will try to open source this project so this is clearer

- the durable objects rate limiter was wholly experimental and didn't make it into production

> All that being said though, maybe all it would've done is prolong the inevitable death due to the product gap the author concludes with.

Very true :(

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#24

Post by zxt_tzx »

smarx007 wrote: post #8
zxt_tzx wrote: post #2

Author here. Over the last few months, I have built and launched a free semantic search tool for GitHub called SemHub ( https://semhub.dev/ ). In this blog post, I share what I’ve learned and why I’ve failed, so that other builders can learn from my experience. This blog post runs long and I have sign-posted each section. I have marked the sections that I consider the particularly insightful with an asterisk (*). I h…

Hi, thanks for building a great tool and a great write-up! I was trying to add a number of repos under oslc/ , oslc-op/ , and eclipse-lyo/* orgs but no joy - internal server error. Hopefully, you will reconsider shutting down the project (just heard about it and am quite excited)! I think a project like yours is going to be helpful to OSS library maintainers to see which features are used in downstream projects and w…

Ohh apologies, I think there was a bug that led to the Internal Server Error, please try again, I _think_ it should be working now!

> I think a project like yours is going to be helpful to OSS library maintainers to see which features are used in downstream projects and which have issues.

That was indeed the original motivation! Will see if I can convince Ammar to reconsider shutting down the project, but no promises

> For this use case, I deployed my own instance to index all OSS repos implementing OSLC REST or using our Lyo SDK

Ohh, in case it's not clear from the UI, you could create an account and index your own "collection" of repos and search from within that interface. I had originally wanted to build out this "collection" concept a lot more (e.g. mixing private and public repos), but I thought it was more important to see if there's traction for the public search idea at all

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#25

Post by zxt_tzx »

gfody wrote: post #21
zxt_tzx wrote: post #2

Author here. Over the last few months, I have built and launched a free semantic search tool for GitHub called SemHub ( https://semhub.dev/ ). In this blog post, I share what I’ve learned and why I’ve failed, so that other builders can learn from my experience. This blog post runs long and I have sign-posted each section. I have marked the sections that I consider the particularly insightful with an asterisk (*). I h…

it's weird you consider this a failure. you spent a few months and learned how to work with embedding models to build an efficient search. the fact that your search works well is a successful outcome. if your goal was to turn a few month effort into a thriving business that's never going to happen period - it only seems possible because when it does happen for people we completely discount the luck factor. if you wan…

> it's weird you consider this a failure. you spent a few months and learned how to work with embedding models to build an efficient search. the fact that your search works well is a successful outcome.

Thank you for your encouragement! I take your point that it was not a technical failure, but I think it's still a product failure in the sense that SemHub was not solving a big enough pain point for sufficiently many people.

> if you want to turn your search into a business now that's a new and different effort, mostly marketing and stuff that most self respecting engineers gives zero shits about, but if that's your real goal don't call it a failure yet because you haven't even tried.

Haha to be honest, my goal was even more modest, SemHub is intended to be a free tool for people to use, we don't intend to monetize it. I also did try to market it (DMing people, Show HN), but the initial users who tried it did not stick around.

Sure, I could've marketed SemHub more, but I think the best ideas carry within themselves a certain virality and I don't think this is it.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#26

Post by zxt_tzx »

romanhn wrote: post #22
zxt_tzx wrote: post #2

Author here. Over the last few months, I have built and launched a free semantic search tool for GitHub called SemHub ( https://semhub.dev/ ). In this blog post, I share what I’ve learned and why I’ve failed, so that other builders can learn from my experience. This blog post runs long and I have sign-posted each section. I have marked the sections that I consider the particularly insightful with an asterisk (*). I h…

Thanks for posting this, very timely as I'm also playing around with pgvector for semantic search. I saw that you ended up trimming inputs longer than 8K tokens. Have you looked into chunking (breaking input into smaller chunks and doing vector search on the chunks)? Embedding models I'm playing with have a max of 512 tokens, so chunking is pretty much a must. Choosing a chunking strategy seems to be a deep rabbit ho…

> Have you looked into chunking (breaking input into smaller chunks and doing vector search on the chunks)?

Ohh I had not seriously considered this until reading this. I could have multiple embeddings per issue and search across those embeddings and if the same issue is matched multiple times, I would probably take the strongest match and dedupe it.

I could create embeddings for comments too and search across those.

Thanks for the suggestion, would be a good think to try!

> Choosing a chunking strategy seems to be a deep rabbit hole of its own.

Yes this is true. In my case, I think the metadata fields like Title and Labels are probably doing a lot of the work (which would be duplicated across chunks?) and, within an issue body, off the top of my head, I can't see any intuitive ways to chunk it.

I have heard that for standard RAG, chunking goes a surprisingly long way!

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#27

Post by zxt_tzx »

johnfn wrote: post #9

That was a great write up. If you don't mind me giving you some unsolicited product feedback: I think SemHub didn't do well because it's unclear what problem it's actually solving. Who actually wants your product? What's the use case? I use GitHub issues all the time, and I can't think of a reason I'd want semhub. If I need to find a particular issue on, say, TypeScript, I'll just google "github typescript issue [des…

Thanks for the feedback, to be honest, my own experience is actually very similar to yours.

The original pain point probably only exists for small minority of open source maintainers who manage multiple repos and actually search across them regularly. Most devs are probably like you and I, and the mediocre GitHub search experience is more than compensated by using Google.

In its current iteration, it's quite hard to get regular devs to change their searching behaviour, and, even for those who experience this pain point, it probably isn't large enough for them to change their behavior.

If I continue to work on this, I would want to (1) solve a bigger + more frequent pain point; (2) build something that requires a smaller change in user behavior.

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#28

Post by zxt_tzx »

nchmy wrote: post #18

This seems pretty similar to something that the ManticoreSearch team released a year ago https://manticoresearch.com/blog/manticoresearch-github-issu... You can index any GH repo and then search it with vector, keyword, hybrid and more. There's faceting and anything else you could ever want. And it is astoundingly fast - even vector search. Here's the direct link to the demo https://github.manticoresearch.com/

oh wow that's super cool, I tried it and it's very fast indeed. thanks for sharing! will spend more time to understand how it's implemented

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#29

Post by zxt_tzx »

serjester wrote: post #14

Great write up, especially agree on pgvector with small (ideally fine tuned) embeddings. There’s so much complexity that comes with keeping your vector db in sync with you main db (especially once you start filtering with metadata). 90% of gen AI apps don’t need it.

> There’s so much complexity that comes with keeping your vector db in sync with you main db (especially once you start filtering with metadata)

Ohh do you speak from experience? I know I will likely never do this, but curious how did you do it? When I looked into this, I found that Airbyte has something to connect the vector db with the main db, but I never bit that bullet (thankfully)

Re: Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

#30

Post by zxt_tzx »

brian-armstrong wrote: post #16

Am I misunderstanding what is meant by semantic code search? I thought the idea was that you run something like a parser on the repo to extract function/class/variable names and then allow searching on a more rich set of data, rather than tokenizing it like English. I know github kind of added this but their version falls apart still even in common languages like C++. It's not unusual for it to just completely miss c…

Ah I was doing semantic search of GitHub _issues_, not the actual code on GitHub.

For code search, I have used grep.app, which works reasonably well

Post reply on HN