RAG was always somewhat of a Frankenstein combining two things that should not be combined: information retrieval based on string matching enhanced with embeddings and LLM that needs not string matching but informative texts. If string matching is good but information is poor or provides wrong context, it would only enforce hallucinations. Search, tool calling and connections should be a part of the system and trained together with LLM
The RAG Obituary: Killed by agents, buried by context windows
131–140 of 185 posts
Re: The RAG Obituary: Killed by agents, buried by context windows
#132This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…
Isn't grep + LLM a form of RAG anyway?
Well yeah RAG just specifies retrieval augmented, not that vector retrieval or decoder retrieval was used
Re: The RAG Obituary: Killed by agents, buried by context windows
#133I get the reasoning behind “letting the agent use grep in a loop“, after all, it is very similar to how humans would explore a document base with ctrl-f. But wouldn’t humans also use vector search all the time if it were as available as ctrl-f? So maybe not ditch vector search but provide it as a tool to the agent. Increased complexity aside, letting the agent explore a huge document base with “vector search in a loo…
Conceptually yes but practically speaking vector search results are generally not sufficiently good
Re: The RAG Obituary: Killed by agents, buried by context windows
#134This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…
Isn't grep + LLM a form of RAG anyway?
Yes, this guy's post came up on my LinkedIn. I think it's helpful to consider the source in these types of articles, written by a CEO at a fintech startup (looks like AI generated too). It's obvious from reading the article that he doesn't understand what he's talking about and has likely never created any kind of RAG or other system. He has a very limited experience, basically a single project, of building a system around rudimentary ingestion of SEC filings, that's his entire breath of technical experience on the subject. So take what you read with a grain of salt, and do your own research and testing.
Re: The RAG Obituary: Killed by agents, buried by context windows
#135This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…
Isn't grep + LLM a form of RAG anyway?
Yeah 100%
Almost all tool calls would result in rag.
Rag is dead just means rolling out your own search and manually injecting results into context is dead (just use tools). It means the chunking techniques are dead.
Re: The RAG Obituary: Killed by agents, buried by context windows
#136This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…
Isn't grep + LLM a form of RAG anyway?
I've seen it called "agentic search" while RAG seems to have become synonymous with semantic search via embeddings
Re: The RAG Obituary: Killed by agents, buried by context windows
#137Earlier quoted context omitted.
Do you trust 'jump to definition'. Obviously it depends on the language server, but it's best effort. I'm often frustrated when it doesn't work, because I broke the code in some way. Or it jumps to a specific definition, but there are multiple. If I was as quick at opening and reading files as claude code, I'd prefer grep with context around the searched term.
In java, for example, jump to definition is pretty flawless.
Unless I'm in an interface and inheritance heavy codebase. Then the first place it jumps to is rarely the one I wanted.
Re: The RAG Obituary: Killed by agents, buried by context windows
#138Earlier quoted context omitted.
Isn't grep + LLM a form of RAG anyway?
I've seen it called "agentic search" while RAG seems to have become synonymous with semantic search via embeddings
That's a silly distinction to make, because there's nothing stopping you from giving an agent access to a semantic search.
If I make a semantic search over my organization's Policy As Code procedures or whatever and give it to Claude Code as an MCP, does Claude Code suddenly stop being agentic?
Re: The RAG Obituary: Killed by agents, buried by context windows
#139Agentic search with a handful of basic tools (drawn from BM25, semantic search, tags, SQL, knowledge graph, and a handful of custom retrieval functions) blows the lid off RAG in my experience. The downside is it takes longer. A single “investigation” can easily use 20-30 different function calls. RAG is like a static one-shot version of this and while the results are inferior the process is also a lot faster.
I've found his hybrid approach pretty good for the majority of use cases. BM25 (maybe Splade if you want a blend of BOW/Keyword), + Vectors + RRF + re-rank works pretty damn well.
The trick that has elevated RAG, at least for my use cases, has been having different representations of your documents, as well as sending multiple permutations of the input query. Do as much as you can in the VectorDB for speed. I'll sometimes have 10-11 different "batched" calls to our vectorDB that are lightning quick. Then also being smart about what payloads I'm actually pulling so that if I do use the LLM to re-rank in the end, I'm not blowing up the context.
TLDR: Yes, you actually do have to put in significant work to build an efficient RAG pipeline, but that's fine and probably should be expected. And I don't think we are in a world yet where we can just "assume" that large context windows will be viable for really precise work, or that costs will drop to 0 anytime soon for those context windows.
Re: The RAG Obituary: Killed by agents, buried by context windows
#140I get the reasoning behind “letting the agent use grep in a loop“, after all, it is very similar to how humans would explore a document base with ctrl-f. But wouldn’t humans also use vector search all the time if it were as available as ctrl-f? So maybe not ditch vector search but provide it as a tool to the agent. Increased complexity aside, letting the agent explore a huge document base with “vector search in a loo…
My mental model (in the "all models are wrong, some are useful" sense) is that vector search is the thing that gives you the terms to grep for.
I like it. With this approach it feels like you also don’t need to fiddle as much with the details of your vector search and DB as that portion just gets you going and the actual retrieval happens with grep in a loop.