Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

21–30 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#21

Post by n_u »

> Reranking: the highest value 5 lines of code you'll add. The chunk ranking shifted a lot. More than you'd expect. Reranking can many times make up for a bad setup if you pass in enough chunks. We found the ideal reranker set-up to be 50 chunk input -> 15 output.

What is re-ranking in the context of RAG? Why not just show the code if it’s only 5 lines?

Re: Production RAG: what I learned from processing 5M+ documents

#22

Post by goleary »

daemonologist wrote: post #8
alexchantavy wrote: post #7

> What moved the needle: Query Generation What does query generation mean in this context, it’s probably not SQL queries right?

It's described in the remainder of the point - they use an LLM to generate additional search queries, either rephrasings of the user's query or bringing additional context from the chat history.

Here's an interesting read on the evolution beyond RAG: https://www.nicolasbustamante.com/p/the-rag-obituary-killed-...

One of the key features in Claude Code is "Agentic Search" aka using (rip)grep/ls to search a codebase without any of the overhead of RAG.

Sounds like even RAG approaches use a similar approach (Query Generation).

Re: Production RAG: what I learned from processing 5M+ documents

#24

Post by tifa2up »

n_u wrote: post #21

> Reranking: the highest value 5 lines of code you'll add. The chunk ranking shifted a lot. More than you'd expect. Reranking can many times make up for a bad setup if you pass in enough chunks. We found the ideal reranker set-up to be 50 chunk input -> 15 output. What is re-ranking in the context of RAG? Why not just show the code if it’s only 5 lines?

OP. Reranking is a specialized LLM that takes the user query, and a list of candidate results, then re-sets the order based on which ones are more relevant to the query.

Here's sample code: https://docs.cohere.com/reference/rerank

Re: Production RAG: what I learned from processing 5M+ documents

#26

Post by underlines »

leetharris wrote: post #14

Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.

rag will be pronounced differently ad again and again. it has its use cases. we moved to agentic search having rag as a tool while other retrieval strategies we added use real time search in the sources. often skipping ingested and chunked soueces. large changes next windows allow for putting almost whole documents into one request.

Re: Production RAG: what I learned from processing 5M+ documents

#28

Post by thethimble »

daemonologist wrote: post #12

I concur: The big LLM-based rerankers (e.g. Qwen3-reranker) are what you always wanted your cross-encoder to be, and I highly recommend giving them a try. Unfortunately they're also quite computationally expensive. Your metadata/tabular data often contains basic facts that a human takes for granted, but which aren't repeated in every text chunk - injecting it can help a lot in making the end model seem less clueless.…

I wish there was more info on the article about actual customer usage - particularly whether it improved process efficiency. It's great to focus on the technical aspects of system optimization but unless this translates to tangible business value it's all just hype.

Re: Production RAG: what I learned from processing 5M+ documents

#29

Post by phillipcarter »

leetharris wrote: post #14

Embedding based RAG will always just be OK at best. It is useful for little parts of a chain or tech demos, but in real life use it will always falter.

Not necessarily? It's been the basis of one of the major ways people would query their data since 2023 on a product I worked on: https://www.honeycomb.io/blog/introducing-query-assistant

The difference is this feature explicitly isn't designed to do a whole lot, which is still the best way to build most LLM-based products and sandwich it between non-LLM stuff.

Post reply on HN