Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

101–110 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#101

Post by PunchTornado »

yahoozoo wrote: post #41
tifa2up wrote: post #24

Earlier quoted context omitted.

OP. Reranking is a specialized LLM that takes the user query, and a list of candidate results, then re-sets the order based on which ones are more relevant to the query. Here's sample code: https://docs.cohere.com/reference/rerank

What is the difference between reranking versus generating text embeddings and comparing with cosine similarity?

the reranker is a cross encoder that sees the docs and the query at the same time. What you normally do is you generating embeddings ahead of time, independent of the prompt used, calculate cosine similarity with the prompt, select the top-k best chunks that match the prompt and only then use a reranker to sort them.

embeddings are a lossy compression, so if you feed the chunks with the prompt at the same time, the results are better. But you can't do this for your whole db, that's why the filtering with cosine similarity at the beginning.

Re: Production RAG: what I learned from processing 5M+ documents

#102

Post by leetharris »

lifty wrote: post #57
leetharris wrote: post #51

Earlier quoted context omitted.

Full text agentic retrieval. Instead of cosine similarity on vectors, parsing metadata through an agentic loop. To give a real world example, the way Claude Code works versus how Cursor's embedded database works.

How do you do that on 5 million documents?

People are usually not querying across 5 million documents in a single scope.

If you want something as simple as "suggest similar tweets" or something across millions of things then embeddings still work.

But if you want something like "compare the documents across these three projects" then you would use full text metadata extraction. Keywords, summaries, table of contents, etc to determine data about each document and each chunk.

Re: Production RAG: what I learned from processing 5M+ documents

#103

Post by googamooga »

pietz wrote: post #98

My biggest RAG learning is to use agentic RAG. (Sorry for buzzword dropping) - Classic RAG: `User -> Search -> LLM -> User` - Agentic RAG: `User LLM Search` Essentially instead of having a fixed loop, you provide the search as a tool to the LLM, which does three things: - The LLM can search multiple times - The LLM can adjust the search query - The LLM can use multiple tools The combination of these three things has…

I fully support this approach! When I first started experimenting—rather naively—with using tool-enabled LLMs to generate documents (such as reports or ADRs) from the extensive knowledge base in Confluence, I built a few tools to help the LLM search Confluence using CQL (Confluence Query Language) and store the retrieved pages in a dedicated folder. The LLM could then search within that folder with simple filesystem tools and pull entire files into its context as needed. The results were quite good, as long as the context didn’t become overloaded. However, when I later tried to switch to a 'Classic RAG' setup, the output quality dropped significantly and I refrained from switching.

Re: Production RAG: what I learned from processing 5M+ documents

#104

Post by DSingularity »

cipherself wrote: post #77
hatmanstack wrote: post #31

Not here to schlep for AWS but S3 Vectors is hands down the SOTA here. That combined with a Bedrock Knowledge Base to handle Discovery/Rebalance tasks makes for the simplest implementation on the Market. Once Bedrock KB backed by S3 Vectors is released from Beta it'll eat everybody's lunch.

S3 Vectors is hands down the SOTA here SOTA for what? Isn't it just a vector store?

I think he just means it should be assumed to be standard practice and considered baseline at this point.

Re: Production RAG: what I learned from processing 5M+ documents

#105

Post by DSingularity »

leetharris wrote: post #14

Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.

Super useful for grounding which is often the only way to robustly protect against hallucinations.

Re: Production RAG: what I learned from processing 5M+ documents

#106

Post by jokethrowaway »

pietz wrote: post #98

My biggest RAG learning is to use agentic RAG. (Sorry for buzzword dropping) - Classic RAG: `User -> Search -> LLM -> User` - Agentic RAG: `User LLM Search` Essentially instead of having a fixed loop, you provide the search as a tool to the LLM, which does three things: - The LLM can search multiple times - The LLM can adjust the search query - The LLM can use multiple tools The combination of these three things has…

yes but the assistant often doesn't search when it should and very rarely does multiple search rounds (both on gpt5 or on claude sonnet 4.5, weaker models are even worse at tool calling)

Re: Production RAG: what I learned from processing 5M+ documents

#107

Post by jokethrowaway »

pietz wrote: post #37

I find it interesting that so many services and tools were investigated except for embedding models. I would have thought that's one of the biggest levers.

i'd go with qwen embedding 3, gemini embeddings or something from mixedbread

Re: Production RAG: what I learned from processing 5M+ documents

#108

Post by pmc00 »

cipherself wrote: post #100
pamelafox wrote: post #89

Earlier quoted context omitted.

Yes, AI Search has a new agentic retrieval feature that includes synthetic query generation: https://techcommunity.microsoft.com/blog/azure-ai-foundry-bl... You can customize the model used and the max # of queries to generate, so latency depends on those factors, plus the length of the conversation history passed in. The model is usually gpt-4o or gpt-4.1 or the -mini of those, so it's the standard latency for those…

Got it, I think this might make sense for a "conversation" type of search not for an instant search feature because lowest latency is gonna be too high IMO.

Fair point on latency, we (Azure AI Search) target both scenarios with different features. For instant search you can just do the usual hybrid + rerank combo, or if you want query rewriting to improve user queries, you can enable QR at a moderate latency hit. We evaluated this approach at length here: https://techcommunity.microsoft.com/blog/azure-ai-foundry-bl...

Of course, agentic retrieval is just better quality-wise for a broader set of scenarios, usual quality-latency trade-off.

We don't do SPLADE today. We've explored it and may get back to it at some point, but we ended up investing more on reranking to boost precision, we've found we have fewer challenges on the recall side.

Re: Production RAG: what I learned from processing 5M+ documents

#109

Post by badlogic »

I run a few production RAG systems, some as old as end of 2023 and arrived at the same conclusions.

Query expansions and non-naive chunking give the biggest bang for the bug, with chunking being the most resource intensive task, if the input data is chunk (pun intended).

Re: Production RAG: what I learned from processing 5M+ documents

#110

Post by cipherself »

DSingularity wrote: post #104
cipherself wrote: post #77

Earlier quoted context omitted.

S3 Vectors is hands down the SOTA here SOTA for what? Isn't it just a vector store?

I think he just means it should be assumed to be standard practice and considered baseline at this point.

Assuming that's what he meant, why would it be considered baseline versus anything else? I am genuinely curious because I'd like to know more about issues people face with this or that vector store in general.
Post reply on HN