Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAG Obituary: Killed by agents, buried by context windows

nicolasbustamante.com

Post reply on HN
Print view View original

111–120 of 185 posts

Re: The RAG Obituary: Killed by agents, buried by context windows

#111

Post by pcthrowaway »

nbstme wrote: post #33
aussieguy1234 wrote: post #25

grep was invented at a time when computers had very small amounts of memory, so small that you might not even be able to load a full text file. So you had tools that would edit one line at a time, or search through a text file one line at a time. LLMs have a similar issue with their context windows. Go back to GPT-2 and you wouldn't have been able to load a text file into its memory. Slowly the memory is increasing,…

Agree. It's a context/memory issue. Soon LLMs will have a 10M context window and they won't need to search. Most codebases are less than 10M tokens.

When dependencies are factored in, I don't know if this is true.

Re: The RAG Obituary: Killed by agents, buried by context windows

#112

Post by madeofpalk »

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

What exactly is RAG? Is it a specific technology, or a technique?

I'm not a super smart AI person, but grepping through a codebase sounds exactly like what RAG is. Isn't tool use just (more sophisticated) RAG?

Re: The RAG Obituary: Killed by agents, buried by context windows

#113

Post by lukaslalinsky »

eru wrote: post #26
themanmaran wrote: post #6

I'm always amazed at claude codes ability to build context by just putting grep in a for loop. It's pretty much the same process I would use in an unfamiliar code base. Just ctrl+f the file system till I find the right starting point.

That's what I used to use as a human, but then I finally overcame my laziness in setting up integration between my editor and compiler (and similar) and got 'jump to definition' working. (Well, I didn't overcome my laziness directly. I just switched from being lazy and not setting up vim and Emacs with the integrations, to trying out vscode where this was trivial or already built in.)

Do you trust 'jump to definition'. Obviously it depends on the language server, but it's best effort. I'm often frustrated when it doesn't work, because I broke the code in some way. Or it jumps to a specific definition, but there are multiple. If I was as quick at opening and reading files as claude code, I'd prefer grep with context around the searched term.

Re: The RAG Obituary: Killed by agents, buried by context windows

#115

Post by eru »

lukaslalinsky wrote: post #113
eru wrote: post #26

Earlier quoted context omitted.

That's what I used to use as a human, but then I finally overcame my laziness in setting up integration between my editor and compiler (and similar) and got 'jump to definition' working. (Well, I didn't overcome my laziness directly. I just switched from being lazy and not setting up vim and Emacs with the integrations, to trying out vscode where this was trivial or already built in.)

Do you trust 'jump to definition'. Obviously it depends on the language server, but it's best effort. I'm often frustrated when it doesn't work, because I broke the code in some way. Or it jumps to a specific definition, but there are multiple. If I was as quick at opening and reading files as claude code, I'd prefer grep with context around the searched term.

> Do you trust 'jump to definition'.

It depends, for some languages 'jump to definition' tools ask the same compiler/interpreter that you use to build your code, so it's as accurate as it gets, and it's not 'best effort'.

It also depends a bit on your project, some project are more prone to re-using names or symbols.

> If I was as quick at opening and reading files as claude code, I'd prefer grep with context around the searched term.

Well, Claude probably also doesn't want to have to 'learn' how to use all kinds of different tools for different languages and eco-systems.

Re: The RAG Obituary: Killed by agents, buried by context windows

#116

Post by larodi »

_the_inflator wrote: post #104
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

I agree. A great many pundits don't get, that RAG means: "a technique that enables large language models (LLMs) to retrieve and incorporate new information" So, RAG is a pattern that is as a principle applied to almost every process. Context windows? Ok, I won't get into all the nitty gritty details here (embedded, small storage device, security, RAM defects, cost and storage of contexts for different contexts etc.),…

Indeed, the name is Retrieval Augmented Generation... so this is generation (synthesis of text) augmented by retrieval (of data from external systems). the goal is to augment the generation, not to improve retrieval.

the improvements needed for the retrieval part are then another topic.

Re: The RAG Obituary: Killed by agents, buried by context windows

#117

Post by lossolo »

rightbyte wrote: post #106
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

I don't get it. Isn't grep RAG?

In RAG, you operate on embeddings and perform vector search, so if you search for fat lady, it might also retrieve text like huge queen, because they're semantically similar. Grep on the other hand, only matches exact strings, so it would not find it.

Re: The RAG Obituary: Killed by agents, buried by context windows

#118

Post by lossolo »

madeofpalk wrote: post #112
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

What exactly is RAG? Is it a specific technology, or a technique? I'm not a super smart AI person, but grepping through a codebase sounds exactly like what RAG is. Isn't tool use just (more sophisticated) RAG?

RAG is a technique, so instead of string matching (like grep), it uses embeddings + vector search to retrieve semantically similar text (car ≈ automobile), then feeds that into the LLM. Tool use is broader, RAG is one pattern within that, but not the same as grep.

Re: The RAG Obituary: Killed by agents, buried by context windows

#119

Post by CuriouslyC »

janalsncm wrote: post #93
nbstme wrote: post #52

Earlier quoted context omitted.

LLMs > rerankers. Yes! I don't like rerankers. They are slow, the context window is small (4096 tokens), it's expensive... It's better when the LLM reads the whole file versus some top_chunks.

Rerankers are orders of magnitude faster and cheaper than LLMs. Typical latency out of the box on a decent sized cross encoder (~4B) will be under 50ms on cheap gpus like an A10G. You won’t be able to run a fancy LLM on that hardware and without tuning you’re looking at hundreds of ms minimum. More importantly, it’s a lot easier to fine tune a reranker on behavior data than an LLM that makes dozens of irrelevant quer…

This is worth emphasizing. At scale, and when you have the resources to really screw around with them to tune your pipeline, rerankers aren't bad, they're just much worse/harder to use out of the box. LLMs buy you easy robustness, baseline quality and capabilities in exchange for cost and latency, which is a good tradeoff until you have strong PMF and you're trying to increase margins.

Re: The RAG Obituary: Killed by agents, buried by context windows

#120

Post by gk1 »

lossolo wrote: post #117
rightbyte wrote: post #106

Earlier quoted context omitted.

I don't get it. Isn't grep RAG?

In RAG, you operate on embeddings and perform vector search, so if you search for fat lady, it might also retrieve text like huge queen, because they're semantically similar. Grep on the other hand, only matches exact strings, so it would not find it.

R in RAG is for retrieval… of any kind. It doesn’t have to be vector search.
Post reply on HN