Similar writeup I did about 1.5 years ago for processing millions of (technical) pages for RAG. Lots has stayed the same it seems
https://jakobs.dev/learnings-ingesting-millions-pages-rag-az...
Production RAG: what I learned from processing 5M+ documents
11–20 of 116 posts
Re: Production RAG: what I learned from processing 5M+ documents
#12I concur:
The big LLM-based rerankers (e.g. Qwen3-reranker) are what you always wanted your cross-encoder to be, and I highly recommend giving them a try. Unfortunately they're also quite computationally expensive.
Your metadata/tabular data often contains basic facts that a human takes for granted, but which aren't repeated in every text chunk - injecting it can help a lot in making the end model seem less clueless.
The point about queries that don't work with simple RAG (like "summarize the most recent twenty documents") is very important to keep in mind. We made our UI very search-oriented and deemphasized the chat, to try to communicate to users that search is what's happening under the hood - the model only sees what you see.
Re: Production RAG: what I learned from processing 5M+ documents
#13Thanks for sharing. TIL about rerankers. Chunking strategy is a big issue. I found acceptable results by shoving large texts to to gemini flash and have it summarize and extract chunks instead of whatever text splitter I tried. I use the method published by Anthropic https://www.anthropic.com/engineering/contextual-retrieval i.e. include full summary along with chunks for each embedding. I also created a tool to enab…
Have you measured your latency, and how sensitive are you to it?
>> Have you measured your latency, and how sensitive are you to it?
Not sensitive to latency at all. My users would rather have well researched answers than poor answers.
Also, I use batch mode APIs for chunking .. it is so much cheaper.
Re: Production RAG: what I learned from processing 5M+ documents
#14Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.
Re: Production RAG: what I learned from processing 5M+ documents
#15I have a RAG setup that doesn't work on documents but other data points that we use for generation (the original data is call recordings but it is heavily processed to just a few text chunks). Instead of a reranker model we do vector search and then simply ask GPT-5 in an extra call which of the results is the most relevant to the input question. Is there an advantage to actual reranker models rather than using a gen…
OP here. rerankers are finetuned small models, they're cheap and very fast compared to an additional GPT-5 call.
It's an async process in my case (custom deep research like) so speed is not that critical
Re: Production RAG: what I learned from processing 5M+ documents
#16They say the chunker is the most important part, but theirs looks rudimentary: https://github.com/agentset-ai/agentset/blob/main/packages/e... That is, there is nothing here that one could not easily write without a library.
I'm not sure there is a chunker in this repo. The file you linked certainly doesn't seem to perform any chunking, it just defines a data model for chunks.
The only place I see that actually operates on chunks does so by fetching them from Redis, and AFAICT nothing in the repo actually writes to Redis, so I assume the chunker is elsewhere.
https://github.com/agentset-ai/agentset/blob/main/packages/j...
Re: Production RAG: what I learned from processing 5M+ documents
#17Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.
What do you recommend? Query generation?
Re: Production RAG: what I learned from processing 5M+ documents
#18Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.
Compared with what?
Re: Production RAG: what I learned from processing 5M+ documents
#19Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.
Most of my ChatGPT queries use RAG (based on the query ChatGPT will decide if it needs to search the web) to get up to date information about the world. In reality life it's effective and it's why every large provider supports it.
Re: Production RAG: what I learned from processing 5M+ documents
#20The point about synthetic query generation is good. We found users had very poor queries, so we initially had the LLM generate synthetic queries. But then we found that the results could vary widely based on the specific synthetic query it generated, so we had it create three variants (all in one LLM call, so that you can prompt it to generate a wide variety, instead of getting three very similar ones back), do parallel search, and then use reciprocal rank fusion to combine the list into a set of broadly strong performers. For the searches we use hybrid dense + sparse bm25, since dense doesn't work well for technical words.
This, combined with a subsequent reranker, basically eliminated any of our issues on search.