Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

81–90 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#81

Post by taneq »

nl wrote: post #75
bityard wrote: post #45

I must be missing something, this says it can be self-hosted. But the first page of the self-hosting docs say you need accounts with no less than 6 (!) other third-party hosted services. We have very different ideas about the meaning of self-hosted.

You can self-host their code. I don't think there is any official definition of "self hosted" that this violates. For example - if a "self hosted" service supports off-site backups is it self hosted or just well designed?

In that case I’m self hosting every web page on the internet because I installed Firefox.

Re: Production RAG: what I learned from processing 5M+ documents

#83

Post by bob_theslob646 »

jweewee wrote: post #82

Does anyone know how to do versioning for embeddings? Let’s say I want to update/upsert my data and deliver v6 of domain data instead of v1 or filter for data within a specified date range. I am thinking of exploring context prepending to chunks.

This is a great question

Re: Production RAG: what I learned from processing 5M+ documents

#85

Post by meander_water »

jweewee wrote: post #82

Does anyone know how to do versioning for embeddings? Let’s say I want to update/upsert my data and deliver v6 of domain data instead of v1 or filter for data within a specified date range. I am thinking of exploring context prepending to chunks.

Your vector store should let you store the original text as well as metadata, where you can store the version. For e.g. turbopuffer lets you filter on attributes https://turbopuffer.com/docs/query#filtering

Re: Production RAG: what I learned from processing 5M+ documents

#87

Post by hawthorns »

yahoozoo wrote: post #41
tifa2up wrote: post #24

Earlier quoted context omitted.

OP. Reranking is a specialized LLM that takes the user query, and a list of candidate results, then re-sets the order based on which ones are more relevant to the query. Here's sample code: https://docs.cohere.com/reference/rerank

What is the difference between reranking versus generating text embeddings and comparing with cosine similarity?

The main point didn't get hit on by the responses. Re-ranking is just a mini-LLM (for latency/cost reasons) that does a double heck. Embedding model finds the closest M documents in R^N space. Re-ranker picks the top K documents from the M documents. In theory, if we just used Gemini 2.5 Pro or GPT 5 as the re-ranker, the performance would even be better than whatever small re-ranker people choose to use.

Re: Production RAG: what I learned from processing 5M+ documents

#89

Post by pamelafox »

cipherself wrote: post #76
pamelafox wrote: post #56

Earlier quoted context omitted.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

I am working on search but rather for text-to-image retrieval, nevertheless, I am curious if by that's all baked into Azure AI search you also meant synthetic query generation from the grandparent comment. If so, what's your latency for this? And do you extract structured data from the query? If so, do you use LLMs for that? Moreover I am curious why you guys use bm25 over SPLADE ?

Yes, AI Search has a new agentic retrieval feature that includes synthetic query generation: https://techcommunity.microsoft.com/blog/azure-ai-foundry-bl... You can customize the model used and the max # of queries to generate, so latency depends on those factors, plus the length of the conversation history passed in. The model is usually gpt-4o or gpt-4.1 or the -mini of those, so it's the standard latency for those. A more recent version of that feature also uses the LLM to dynamically decide which of several indices to query, and executes the searches in parallel.

That query generation approach does not extract structured data. I do maintain another RAG template for PostgreSQL that uses function calling to turn the query into a structured query, such that I can construct SQL filters dynamically. Docs here: https://github.com/Azure-Samples/rag-postgres-openai-python/...

I'll ask the search about SPLADE, not sure.

Re: Production RAG: what I learned from processing 5M+ documents

#90

Post by kkapelon »

nl wrote: post #75
bityard wrote: post #45

I must be missing something, this says it can be self-hosted. But the first page of the self-hosting docs say you need accounts with no less than 6 (!) other third-party hosted services. We have very different ideas about the meaning of self-hosted.

You can self-host their code. I don't think there is any official definition of "self hosted" that this violates. For example - if a "self hosted" service supports off-site backups is it self hosted or just well designed?

> For example - if a "self hosted" service supports off-site backups is it self hosted or just well designed?

There is a big difference between communicating with external services (your example) vs REQUIRING external services (what parent is complaining about).

If in your example the system can run correctly with just local backups I would consider it self-hosted.

Post reply on HN