Production RAG: what I learned from processing 5M+ documents
blog.abdellatif.io
Production RAG: what I learned from processing 5M+ documents
1–10 of 116 posts
Re: Production RAG: what I learned from processing 5M+ documents
#2Thanks for sharing. TIL about rerankers.
Chunking strategy is a big issue. I found acceptable results by shoving large texts to to gemini flash and have it summarize and extract chunks instead of whatever text splitter I tried. I use the method published by Anthropic https://www.anthropic.com/engineering/contextual-retrieval i.e. include full summary along with chunks for each embedding.
I also created a tool to enable the LLM to do vector search on its own .
I do not use Langchain or python.. I use Clojure+ LLMs' REST APIs.
Re: Production RAG: what I learned from processing 5M+ documents
#3I have a RAG setup that doesn't work on documents but other data points that we use for generation (the original data is call recordings but it is heavily processed to just a few text chunks).
Instead of a reranker model we do vector search and then simply ask GPT-5 in an extra call which of the results is the most relevant to the input question. Is there an advantage to actual reranker models rather than using a generic LLM?
Re: Production RAG: what I learned from processing 5M+ documents
#4I have a RAG setup that doesn't work on documents but other data points that we use for generation (the original data is call recordings but it is heavily processed to just a few text chunks). Instead of a reranker model we do vector search and then simply ask GPT-5 in an extra call which of the results is the most relevant to the input question. Is there an advantage to actual reranker models rather than using a gen…
OP here. rerankers are finetuned small models, they're cheap and very fast compared to an additional GPT-5 call.
Re: Production RAG: what I learned from processing 5M+ documents
#5They say the chunker is the most important part, but theirs looks rudimentary: https://github.com/agentset-ai/agentset/blob/main/packages/e...
That is, there is nothing here that one could not easily write without a library.
Re: Production RAG: what I learned from processing 5M+ documents
#6Thanks for sharing. TIL about rerankers. Chunking strategy is a big issue. I found acceptable results by shoving large texts to to gemini flash and have it summarize and extract chunks instead of whatever text splitter I tried. I use the method published by Anthropic https://www.anthropic.com/engineering/contextual-retrieval i.e. include full summary along with chunks for each embedding. I also created a tool to enab…
Have you measured your latency, and how sensitive are you to it?
Re: Production RAG: what I learned from processing 5M+ documents
#7> What moved the needle: Query Generation
What does query generation mean in this context, it’s probably not SQL queries right?
Re: Production RAG: what I learned from processing 5M+ documents
#8> What moved the needle: Query Generation What does query generation mean in this context, it’s probably not SQL queries right?
It's described in the remainder of the point - they use an LLM to generate additional search queries, either rephrasings of the user's query or bringing additional context from the chat history.
Re: Production RAG: what I learned from processing 5M+ documents
#9They say the chunker is the most important part, but theirs looks rudimentary: https://github.com/agentset-ai/agentset/blob/main/packages/e... That is, there is nothing here that one could not easily write without a library.
OP here. We've been working on agentset.ai full-time for 2 months. The product now gets you something working quite well out of the box. Better than most people with no experience in RAG (I'd say so with confidence).
Ingestion + Agentic Search are two areas that we're focused on in the short term.
Re: Production RAG: what I learned from processing 5M+ documents
#10Exactly what kind of processing was done? Your pipeline is a function of the use case, lest you overengineer…