tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.
Pruning RAG context down to what the answer actually needs
31–40 of 54 posts
Re: Pruning RAG context down to what the answer actually needs
#32"Three knobs matter:" I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.
Re: Pruning RAG context down to what the answer actually needs
#33Re: Pruning RAG context down to what the answer actually needs
#34Re: Pruning RAG context down to what the answer actually needs
#35Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when refe…
Re: Pruning RAG context down to what the answer actually needs
#36Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when refe…
RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context. The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever. This does not…
So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.
Re: Pruning RAG context down to what the answer actually needs
#37Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when refe…
If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.
If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.
Re: Pruning RAG context down to what the answer actually needs
#38Earlier quoted context omitted.
RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hea…
I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it? The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval Also is i…
You need a lot more unstructured data than most typical “RAG” users doing document search are dealing with for it it to not be a solved problem, IMO (just give a tool calling agent your sql schema/directory structure). Even that is still an interesting problem for more typical use cases, but only at large scales where you start needing to do multiple passes or fan-out or convert data that could be structured like that into data that already is. I’m interested in large scale code search, coding agent context/conversation search, and network/trace analysis which has a lot of domain-specific considerations that make it interesting but definitely not structured like a typical “document chunking with cosine similarity” RAG implementation.
Re: Pruning RAG context down to what the answer actually needs
#39i wouldn't trust the small llm there. it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!