> LLM: GPT 4.1 -> GPT 5 -> GPT 4.1, covered by Azure credits
whats this roundtrip? also the chronology of the LLM (4.1) doesnt match the rest of the stack (text-embedding-large-3), feels weird
Production RAG: what I learned from processing 5M+ documents
91–100 of 116 posts
Re: Production RAG: what I learned from processing 5M+ documents
#92To somebody thinking of building or paying for such a RAG system, would a workable solution be:
* Upload documents via API into a Google Workspace folder * Use some sort of Google AI search API on those documents in that folder
…placing documents for different customers into different folders.
Or the Azure equivalent whatever that is.
Re: Production RAG: what I learned from processing 5M+ documents
#93> LLM: GPT 4.1 -> GPT 5 -> GPT 4.1, covered by Azure credits whats this roundtrip? also the chronology of the LLM (4.1) doesnt match the rest of the stack (text-embedding-large-3), feels weird
OP. We migrated to GPT-5 when it came out but found that it performs worse than 4.1 when you pass lots of context (up to 100K tokens in some cases). We found that it:
a) has worse instruction following; doesn't follow the system prompt b) produces very long answers which resulted in a bad ux c) has 125K context window so extreme cases resulted in an error
Again, these were only observed in RAG when you pass lots of chunks, GPT-5 is probably a better model for other taks.
Re: Production RAG: what I learned from processing 5M+ documents
#94The point about synthetic query generation is good. We found users had very poor queries, so we initially had the LLM generate synthetic queries. But then we found that the results could vary widely based on the specific synthetic query it generated, so we had it create three variants (all in one LLM call, so that you can prompt it to generate a wide variety, instead of getting three very similar ones back), do paral…
Yep- that's all best practice. I want to know if we could push performance further- routing the query to different embedding models or scoring strategies, or using multiple re-rankers- still feels like the process is missing something.
OP. The way you improve it is move away from single shot semantic/keyword search and have an agentic system that can evaluate results and do follow-up queries.
Re: Production RAG: what I learned from processing 5M+ documents
#95How much of a hit would you take on quality if you moved the processing local? have you experimented with it? don’t think llamaindex has local sadly
Quite a decent hit. Local models don't perform very well in long contexts. We're planning to support a local-only offline set-up for people to host w/o additional dependencies
Re: Production RAG: what I learned from processing 5M+ documents
#96Anybody know what is meant by 'injecting relevant metadata'. Where is it injected?
You typically add a lot of metadata with each chunk text to be able to filter it, and do to include in the citations. Injecting metadata means that you see what metadata adds helpful context to the LLM, and when you pass the results to the LLM you pass them in a format like this:
Title: ... Author: ... Text: ...
for each chunk, instead of just passing the text
Re: Production RAG: what I learned from processing 5M+ documents
#97Great post, gonna be super useful for me :)
Re: Production RAG: what I learned from processing 5M+ documents
#98My biggest RAG learning is to use agentic RAG. (Sorry for buzzword dropping)
- Classic RAG: `User -> Search -> LLM -> User`
- Agentic RAG: `User LLM Search`
Essentially instead of having a fixed loop, you provide the search as a tool to the LLM, which does three things:
- The LLM can search multiple times
- The LLM can adjust the search query
- The LLM can use multiple tools
The combination of these three things has solved a majority of classic RAG problems. It improves user queries, it can map abbreviations, it can correct bad results on its own, you can also let it list directories and load files directly.
Re: Production RAG: what I learned from processing 5M+ documents
#99Earlier quoted context omitted.
Boy, that should not be the concern of the end user (developer) but those implementing RAG solutions as a service at Amazon, Microsoft, Openai and so on.
At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…
"It's also possible to do it on top of other DBs like Postgres, but takes more effort."
Shameless plug: plpgsql_bm25: BM25 search implemented in PL/pgSQL (The Unlicense / PUBLIC DOMAIN)
https://github.com/jankovicsandras/plpgsql_bm25
There's an example Postgres_hybrid_search_RRF.ipynb in the repo which shows hybrid search with Reciprocal Rank Fusion ( plpgsql_bm25 + pgvector ).
Re: Production RAG: what I learned from processing 5M+ documents
#100Earlier quoted context omitted.
I am working on search but rather for text-to-image retrieval, nevertheless, I am curious if by that's all baked into Azure AI search you also meant synthetic query generation from the grandparent comment. If so, what's your latency for this? And do you extract structured data from the query? If so, do you use LLMs for that? Moreover I am curious why you guys use bm25 over SPLADE ?
Yes, AI Search has a new agentic retrieval feature that includes synthetic query generation: https://techcommunity.microsoft.com/blog/azure-ai-foundry-bl... You can customize the model used and the max # of queries to generate, so latency depends on those factors, plus the length of the conversation history passed in. The model is usually gpt-4o or gpt-4.1 or the -mini of those, so it's the standard latency for those…
Got it, I think this might make sense for a "conversation" type of search not for an instant search feature because lowest latency is gonna be too high IMO.