[flagged]
Pruning RAG context down to what the answer actually needs
11–20 of 54 posts
Re: Pruning RAG context down to what the answer actually needs
#12[deleted]
Re: Pruning RAG context down to what the answer actually needs
#13Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when refe…
I read “RAG Context” as “the retrieved content injected into the context window”
So when an agent does "cat file.txt" that's RAG to you?
Re: Pruning RAG context down to what the answer actually needs
#14tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.
Re: Pruning RAG context down to what the answer actually needs
#15Earlier quoted context omitted.
I read “RAG Context” as “the retrieved content injected into the context window”
So when an agent does "cat file.txt" that's RAG to you?
The answer would be yes. It's about using stored knowledge to increase the accuracy of the answer and evidence surfacing.
It doesn't have to be a vector database.
Kapa is one of the few companies doing RAG right.
Re: Pruning RAG context down to what the answer actually needs
#16tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.
That's what the rubric is for; it reduces the problem to NLP, which is the LLM's forte. The more objective you can make your rubric, the better.
Re: Pruning RAG context down to what the answer actually needs
#17Earlier quoted context omitted.
I read “RAG Context” as “the retrieved content injected into the context window”
So when an agent does "cat file.txt" that's RAG to you?
It is to me. And I agree that the term is losing value because it's becoming ubiquitous but it's the differentiation from the first versions of ChatGPT etc., which were purely user input -> LLM -> output driven.
Re: Pruning RAG context down to what the answer actually needs
#18I think this is really where energy saving comes into play. Context is so incredibly energy and processing time sensitive.
Re: Pruning RAG context down to what the answer actually needs
#19Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks). "Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social med…
"RAG is dead" was declared because of ever expanding context windows.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.
Re: Pruning RAG context down to what the answer actually needs
#20Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when refe…
RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.