Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAG Obituary: Killed by agents, buried by context windows

nicolasbustamante.com

Post reply on HN
Print view View original

51–60 of 185 posts

Re: The RAG Obituary: Killed by agents, buried by context windows

#51

Post by jgalt212 »

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

> Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use cases involve millions of documents across distributed systems

Great point, but this grep in a loop probably falls apart (i.e. becomes non-performant) at 1000s of docs, not millions and 10s of simultaneous users

Re: The RAG Obituary: Killed by agents, buried by context windows

#52

Post by nbstme »

CuriouslyC wrote: post #45
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

Agentic retrieval is really more a form of deep research (from a product standpoint there is very little difference). The key is that LLMs > rerankers, at least when you're not at webscale where the cost differential is prohibitive.

LLMs > rerankers. Yes! I don't like rerankers. They are slow, the context window is small (4096 tokens), it's expensive... It's better when the LLM reads the whole file versus some top_chunks.

Re: The RAG Obituary: Killed by agents, buried by context windows

#53

Post by dymk »

tptacek wrote: post #48
serf wrote: post #35

Earlier quoted context omitted.

in the same vein that a 'Behind The Scenes Look At The Making of Jurassic Park' is , in fact, an ad. having a company name pitched at you within the first two sentences is a pretty good give away.

3/4 of what hits the front page is an "ad" by that standard. I don't see how you can get less promotional than a long-form piece about why your tech is obsolete. Seems just mean-spirited.

It’s because the article’s main goal is to sell me the company’s product, not inform me about RAG. It’s a zero calorie article.

Re: The RAG Obituary: Killed by agents, buried by context windows

#54

Post by nbstme »

jgalt212 wrote: post #51
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

> Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use cases involve millions of documents across distributed systems Great point, but this grep in a loop probably falls apart (i.e. becomes non-performant) at 1000s of docs, not millions and 10s of simultaneous users

Why does grep in a loop fall apart? It’s expensive, sure, but LLM costs are trending toward zero. With Sonnet 4.5, we’ve seen models get better at parallelization and memory management (compacting conversations and highlighting findings).

Re: The RAG Obituary: Killed by agents, buried by context windows

#55

Post by nbstme »

jgalt212 wrote: post #49

I'm not feeling it. Constantly pinging these yuge LLMs is not economic and not good for sensitive docs.

But don’t you think LLM pricing is heading toward zero? It seems to halve every six months. And on privacy, you can hope model providers won’t train on your data, (but there’s no guarantee)

Re: The RAG Obituary: Killed by agents, buried by context windows

#56

Post by nbstme »

dymk wrote: post #14
thenewwazoo wrote: post #3

[flagged]

I don’t mind articles that have a hint of “an AI helped write this” as long as the content is actually informationally dense and well explained. But this article is an obvious ad, has almost no interesting information or summaries or insights, and has the… weirdly chipper? tone that AI loves to glaze readers with.

Why call it an ad? It’s not even on the company site. I only mentioned my company upfront so people get context (why we had to build a complex RAG pipeline, what kinds of documents we’re working with, and why the examples come from real production use cases).

Re: The RAG Obituary: Killed by agents, buried by context windows

#57

Post by nbstme »

tptacek wrote: post #48
serf wrote: post #35

Earlier quoted context omitted.

in the same vein that a 'Behind The Scenes Look At The Making of Jurassic Park' is , in fact, an ad. having a company name pitched at you within the first two sentences is a pretty good give away.

3/4 of what hits the front page is an "ad" by that standard. I don't see how you can get less promotional than a long-form piece about why your tech is obsolete. Seems just mean-spirited.

haha so true!

Re: The RAG Obituary: Killed by agents, buried by context windows

#58

Post by nbstme »

momojo wrote: post #15
thenewwazoo wrote: post #3

[flagged]

I'm guessing first draft was AI. I had to re-read that part a couple times because the flow was off. That second paragraph was completely unnecessary too since the previous paragraph already got the point across that "context window small in 2022". On the whole though, I still learned a lot.

Thanks! Sorry if the flow was off

Re: The RAG Obituary: Killed by agents, buried by context windows

#59

Post by nbstme »

dkga wrote: post #44

RAG is the new US dollar, now every year someone will predict its looming death…

HAHAHA. Ok let's call it "transformation." As i wrote "The next decade of AI search will belong to systems that read and reason end-to-end. Retrieval isn’t dead—it’s just been demoted."

Re: The RAG Obituary: Killed by agents, buried by context windows

#60

Post by voidhorse »

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

Not to mention, unless you want to ship entire containers, you are beholden to the unknown quirks of tools on whatever system your agent happens to execute on. It's like taking something already nondeterministic and extremely risky and ceding even more control—let's all embrace chaos.

Generative AI is here to stay, but I have a feeling we will look back on this period of time in software engineering as a sort of dark age of the discipline. We've seemingly decided to abandon almost every hard won insight and practice about building robust and secure computational systems overnight. It's pathetic that this industry so easily sold itself to the illogical sway of marketers and capital.

Post reply on HN