Pruning RAG context down to what the answer actually needs

i wouldn't trust the small llm there.

it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!

I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it?

The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval

Also is it just me that doesn't like this sort of wording?

> agent has to hold

I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context

[flagged]

RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context.

The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.

This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.

I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.

The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.

What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.

But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).

The contribution here is the technique (which looks like a nice alternative to reranking) rather than the article or it being some kind of think piece, so does it really matter?

I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.