Show HN: Building a web search engine from scratch with 3B neural embeddings • Hacker News Board

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#51

Post by arkmm » Tue, Aug 12, 2025, 11:36 PM UTC

"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 100× more expensive, to say nothing of other APIs."

I wonder if OpenAI uses this as a honeypot to get domain-specific source data into its training corpus that it might otherwise not have access to.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#52

Post by ww520 » Tue, Aug 12, 2025, 11:41 PM UTC

Mad respect. This is an incredible project to pull together all these technologies. The crown jewel of a search engine is its ranking algorithm. I'm not sure how LLM is being used in this regard in here.

One effective old technique for ranking is to capture the search-to-click relationship by real users. It's basically the training data by human mapping the search terms they entered to the links they clicked. With just a few of clicks, the ranking relevance goes way up.

May be feeding the data into a neural net would help ranking. It becomes a classification problem - given these terms, which links have higher probabilities being clicked. More people clicking on a link for a term would strengthening the weights.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#53

Post by costco » Wed, Aug 13, 2025, 12:05 AM UTC

This is awesome, and the low cost is especially impressive. I rarely have the motivation after working on a side project to actually document all the decisions made along the way, much less in such a thorough way. Regarding your CoreNN library, Clearview has a blog post [1] on how they index 30 billion face embeddings that you may find interesting. They combine RocksDB with faiss.

[1] https://www.clearview.ai/post/how-we-store-and-search-30-bil...

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#54

Post by [deleted] » Wed, Aug 13, 2025, 12:13 AM UTC

arkmm wrote: post #51

"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…

[deleted]

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#55

Post by mfrye0 » Wed, Aug 13, 2025, 12:17 AM UTC

This was super cool to read. I'm developing something somewhat similar, but for business search, and ran into a lot of the same challenges. Everyone thinks crawling, processing, and indexing data is easy, but doing it cost effectively at scale is a completely different beast.

Kudos wilsonzlin. I'd love to chat sometime if you see this. It's a small space of people that can build stuff like this e2e.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#56

Post by osigurdson » Wed, Aug 13, 2025, 1:09 AM UTC

jvia wrote: post #49

A vector-only search engine will fail for a lot of common use cases where the keywords do matter. I tried searching for `garbanzo bean stew` and got totally irrelevant bean recipes.

Agree. For best results both lexical and vector search results should be fed into a reranker. Slow and expensive but high quality.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#57

Post by cedws » Wed, Aug 13, 2025, 1:26 AM UTC

arkmm wrote: post #51

"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 10…

I don’t think OpenAI train on data processed via the API, unless there’s an exception specifically for this.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#58

Post by ryanlime » Wed, Aug 13, 2025, 1:27 AM UTC

One of the most insightful posts I’ve read recently. I especially enjoy the rationale behind the options you chose to reduce costs and going into detail on where you find the most savings.

I know the post primarily focuses on neural search, but I’m wondering you tried integrating hybrid BM-25 + embeddings search and if this led to any improvements. Also, what reranking models did you find most useful and cost efficient?

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#59

Post by dcreater » Wed, Aug 13, 2025, 1:29 AM UTC

This wasn't even in the realm of what I thought is possible for a single person to do. Incredible work!

It doesn't seem that far in diatance from a commercial search engine? Maybe even Google?

50k to run is a comically small number. I'm tempted to just give you that money to seed.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#60

Post by seriocomic » Wed, Aug 13, 2025, 1:53 AM UTC

Adding my kudos to the other commenters here - the polymath skills necessary to take on something like this is remarkable as a solo effort. I was hoping for more detail on the issues found during the request/parsing at a domain/page level.