The RAG Obituary: Killed by agents, buried by context windows • Hacker News Board

Re: The RAG Obituary: Killed by agents, buried by context windows

#141

Post by hannasanarion » Thu, Oct 02, 2025, 1:13 PM UTC

peab wrote: post #135

lemonlearnings wrote: post #130

Earlier quoted context omitted.

Isn't grep + LLM a form of RAG anyway?

Yeah 100% Almost all tool calls would result in rag. Rag is dead just means rolling out your own search and manually injecting results into context is dead (just use tools). It means the chunking techniques are dead.

Chunking is still relevant, because you want your tool calls to return results specific to the needs of the query.

If you want to know "how are tartans officially registered" you don't want to feed the entire 554kb wikipedia article on Tartan to your model, using 138,500 tokens, over 35% of gpt-5's context window, with significant monetary and latency cost. You want to feed it just the "Regulation>Registration" subsection and get an answer 1000x cheaper and faster.

Re: The RAG Obituary: Killed by agents, buried by context windows

#142

Post by tedivm » Thu, Oct 02, 2025, 1:14 PM UTC

lemonlearnings wrote: post #130

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

Isn't grep + LLM a form of RAG anyway?

It really depends on what you mean by RAG. If you take the acronym at face value yeah.

However, RAG has been used as a stand in for a specific design pattern where you retrieve data at the start of a conversation or request and then inject that into the request. This simple pattern has benefits compared to just using sending a prompt by itself.

The point the author is trying to make is that this pattern kind of sucks compared to Agentic Search, where instead of shoving a bunch of extra context in at the start you give the model the ability to pull context in as needed. By switching from a "push" to a "pull" pattern, we allow the model to augment and clarify the queries it's making as it goes through a task which in turn gives the model better data to work with (and thus better results).

Re: The RAG Obituary: Killed by agents, buried by context windows

#143

Post by hannasanarion » Thu, Oct 02, 2025, 1:15 PM UTC

lossolo wrote: post #121

gk1 wrote: post #120

Earlier quoted context omitted.

R in RAG is for retrieval… of any kind. It doesn’t have to be vector search.

Sure, but vector search is the dominant form of RAG, the rest are niche. Saying "RAG doesn’t have to use vectors" is like saying "LLMs don't have to use transformers". Technically true, but irrelevant when 99% of what's in use today does.

How are they niche? The default mode of search for most dedicated RAG apps nowadays is hybrid search that blends classical BM-25 search with some HNSW embedding search. That's already breaking the definition.

A search is a search. The architecture doesn't care if it's doing an vector search or a text search or a keyword search or a regex search, it's all the same. Deploying a RAG app means trying different search methods, or using multiple methods simultaneously or sequentially, to get the best performance for your corpus and use case.

Re: The RAG Obituary: Killed by agents, buried by context windows

#144

Post by hannasanarion » Thu, Oct 02, 2025, 1:18 PM UTC

madeofpalk wrote: post #112

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

What exactly is RAG? Is it a specific technology, or a technique? I'm not a super smart AI person, but grepping through a codebase sounds exactly like what RAG is. Isn't tool use just (more sophisticated) RAG?

Yes, you are right. The OP has a weirdly narrow definition of what RAG is.

Only the most basic "hello world" type RAG systems rely exclusively on vector search. Everybody has been doing hybrid search or multiple simultaneous searches exposed through tools for quite some time now.

Re: The RAG Obituary: Killed by agents, buried by context windows

#145

Post by anshumankmr » Thu, Oct 02, 2025, 1:34 PM UTC

Honestly, I am not sure when RAG had its heyday to be deserving an obituary. I still think that it is in its early days.

Re: The RAG Obituary: Killed by agents, buried by context windows

#146

Post by mk_stjames » Thu, Oct 02, 2025, 1:35 PM UTC

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

This was essentially my response as well, but the other replies to you also have a point, and I think the key here is the 'Retrieval' in RAG is very vague, and depending on who you were and what you were getting into RAG for, the term means different things.

I am definitely more aligned with needing what I would rather call 'Deep Semantic Search and Generation' - the ability to query text chunk embeddings of... a 100k PDF's, using the semantics to search for the closeness of the 'ideas', those fed into the context of the LLM, and then the LLM generate a response to the prompt citing the source PDF(s) the closest matched vectors came from...

That is the killer app of a 'deep research' assistant IMO and you don't get that via just grepping words and feeding related files into the context window.

The downside is, how to generate embeddings of massive amounts of mixed-media files and store in a database quickly and cheaply compared to just grepping a few terms from said files? A CPU grep of text in files in RAM is like five orders of magnitude faster than an embedding model on the GPU generating semantic embeddings of the chunked file and then storing those for later.

Re: The RAG Obituary: Killed by agents, buried by context windows

#147

Post by deepsquirrelnet » Thu, Oct 02, 2025, 1:40 PM UTC

janalsncm wrote: post #93

nbstme wrote: post #52

Earlier quoted context omitted.

LLMs > rerankers. Yes! I don't like rerankers. They are slow, the context window is small (4096 tokens), it's expensive... It's better when the LLM reads the whole file versus some top_chunks.

Rerankers are orders of magnitude faster and cheaper than LLMs. Typical latency out of the box on a decent sized cross encoder (~4B) will be under 50ms on cheap gpus like an A10G. You won’t be able to run a fancy LLM on that hardware and without tuning you’re looking at hundreds of ms minimum. More importantly, it’s a lot easier to fine tune a reranker on behavior data than an LLM that makes dozens of irrelevant quer…

More than that, adding longer context isn’t free either in time or money. So filling an LLM context with k=100 documents of mixed relevance may be slower than reranking and filling with k=10 of high relevance.

Of course, the devil is in the details and there’s five dozen reasons why you might choose one approach over the other. But it is not clear that using a reranker is always slower.

Re: The RAG Obituary: Killed by agents, buried by context windows

#148

Post by codyb » Thu, Oct 02, 2025, 2:22 PM UTC

cmenge wrote: post #64

We're processing tenders for the construction industry - this comes with a 'free' bucket sort from the start, namely that people practically always operate only on a single tender. Still, that single tender can be on the order of a billion tokens. Even if the LLM supported that insane context window, it's roughly 4GB that need to be moved and with current LLM prices, inference would be thousands of dollars. I detaile…

For anyone unfamiliar, construction tenders are part of the project bidding process and appear to be a structured and formal manner in which contractors submit bids for large projects.

Re: The RAG Obituary: Killed by agents, buried by context windows

#149

Post by codyb » Thu, Oct 02, 2025, 2:23 PM UTC

eru wrote: post #26

themanmaran wrote: post #6

I'm always amazed at claude codes ability to build context by just putting grep in a for loop. It's pretty much the same process I would use in an unfamiliar code base. Just ctrl+f the file system till I find the right starting point.

That's what I used to use as a human, but then I finally overcame my laziness in setting up integration between my editor and compiler (and similar) and got 'jump to definition' working. (Well, I didn't overcome my laziness directly. I just switched from being lazy and not setting up vim and Emacs with the integrations, to trying out vscode where this was trivial or already built in.)

Jump to definition works when you have the starting point already.

I use both grep and JTD fairly frequently for different use cases.

Re: The RAG Obituary: Killed by agents, buried by context windows

#150

Post by codyb » Thu, Oct 02, 2025, 2:24 PM UTC

lukaslalinsky wrote: post #113

eru wrote: post #26

Earlier quoted context omitted.

That's what I used to use as a human, but then I finally overcame my laziness in setting up integration between my editor and compiler (and similar) and got 'jump to definition' working. (Well, I didn't overcome my laziness directly. I just switched from being lazy and not setting up vim and Emacs with the integrations, to trying out vscode where this was trivial or already built in.)

Do you trust 'jump to definition'. Obviously it depends on the language server, but it's best effort. I'm often frustrated when it doesn't work, because I broke the code in some way. Or it jumps to a specific definition, but there are multiple. If I was as quick at opening and reading files as claude code, I'd prefer grep with context around the searched term.

In ViM with the CoC code completion plugin JTD gives me multiple options when there are many and I can choose the sensible one in a popover before the actual jump occurs.

I believe that was my experience with IDEs too?