Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAG Obituary: Killed by agents, buried by context windows

nicolasbustamante.com

Post reply on HN
Print view View original

81–90 of 185 posts

Re: The RAG Obituary: Killed by agents, buried by context windows

#81

Post by ako »

sergiotapia wrote: post #40

>The winners will not be the ones who maintain the biggest vector databases, but the ones who design the smartest agents to traverse abundant context and connect meaning across documents. So if one were building say a memory system for an AI chat bot, how would you save all the data related to a user? Mother's name, favorite meals, allergies? If not a Vector database like pinecone, then what? Just a big .txt file per…

That is what Claude Sonnet 4.5 is doing: https://youtu.be/pidnIHdA1Y8?si=GqNEYBFyF-3Klh4-

Re: The RAG Obituary: Killed by agents, buried by context windows

#82

Post by imiric »

nbstme wrote: post #55
jgalt212 wrote: post #49

I'm not feeling it. Constantly pinging these yuge LLMs is not economic and not good for sensitive docs.

But don’t you think LLM pricing is heading toward zero? It seems to halve every six months. And on privacy, you can hope model providers won’t train on your data, (but there’s no guarantee)

Oh, it's going to be "free" alright, in the same way that most web services are today. I.e., you will pay for it with your data and attention.

The only difference is that the advertising will be much more insidious and manipulative, the data collection far easier since people are already willingly giving it up, and the business much more profitable.

I can hardly wait.

Re: The RAG Obituary: Killed by agents, buried by context windows

#83

Post by delusional »

Analemma_ wrote: post #34
nbstme wrote: post #28

Earlier quoted context omitted.

It's mind blowing. It's so simple, elegant and... effective! Grep+glob and a lot of iterations is all we need.

We always suspected find+grep+xargs was Turing-complete, and now Claude is proving it.

That's one of the most nonsensical comments on all of hackernews. A Markov change could have wrote it.

What do you mean Turing complete? Obviously all 3 programs are running on a Turing complete machine. Xargs is a runner for other commands, obviously those commands can be Turing complete.

I haven't heard of anybody working on a _proof_ for the Turing completeness of xargs, and I think the only conference willing to publish it would be Sigbovik.

Re: The RAG Obituary: Killed by agents, buried by context windows

#84

Post by SV_BubbleTime »

nbstme wrote: post #41
Analemma_ wrote: post #34

Earlier quoted context omitted.

We always suspected find+grep+xargs was Turing-complete, and now Claude is proving it.

Exactly. AGI implies minimal tooling and very primitive tools.

AGI implies that a system is financially viable to let run 24 hours a day with little to no direction.

No amount of find+grep+LLM is even remotely there yet.

Re: The RAG Obituary: Killed by agents, buried by context windows

#85

Post by SV_BubbleTime »

tptacek wrote: post #48
serf wrote: post #35

Earlier quoted context omitted.

in the same vein that a 'Behind The Scenes Look At The Making of Jurassic Park' is , in fact, an ad. having a company name pitched at you within the first two sentences is a pretty good give away.

3/4 of what hits the front page is an "ad" by that standard. I don't see how you can get less promotional than a long-form piece about why your tech is obsolete. Seems just mean-spirited.

> 3/4 of what hits the front page is an "ad" by that standard.

Is anyone disagreeing with that?

Re: The RAG Obituary: Killed by agents, buried by context windows

#87

Post by devmor »

This reads like someone AI-generated prose to defend something they want to invest in and decry something it competes with. It does not come off as honest, written by a human, or useful to anyone outside of the specific, narrow contexts the "author" sees for the technologies mentioned.

Frankly, reading through this at makes me feel as though I am a business analyst or engineering manager being presented with a project proposal from someone very worried that a competing proposal will take away their chance to shine.

As it reaches the end, I feel like I'm reading the same thing, but presented to a Buzzfeed reader.

Re: The RAG Obituary: Killed by agents, buried by context windows

#88

Post by glenngillen »

nbstme wrote: post #50
davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

Appreciate the feedback. I’m not saying grep replaces RAG. The shift is that bigger context windows let LLMs just read whole files, so you don’t need the whole chunk/embed pipeline anymore. Grep is just a quick way to filter down candidates. From there the model can handle 100–200 full docs and jot notes into a markdown file to stay within context. That’s a very different workflow than classic RAG.

I was previously working at https://autonomy.computer, and building out a platform for autonomous products (i.e., agents) there. I started to observe a similar opportunity. We had an actor-based approach to concurrency that meant it was super cheap performance-wise to spin up a new agent. _That_ in turn meant a lot of problems could suddenly become embarrassingly parallel, and that rather than pre-computing/caching a bunch of stuff into a RAG system you could process whatever you needed in a just-in-time approach. List all the documents you've got, spawn a few thousand agents and give each a single document to process, aggregate/filter the relevant answers when they come back.

Obviously that's not the optimal approach for every use case, but there's a lot where IMO it was better. In particular I was hoping to spend more time exploring it in an enterprise context where you've got complicated sharing and permission models to take into consideration. If you have agents simply passing through the permission of the user executing the search whatever you get back is automatically constrained to only the things they had access to in that moment. As opposed to other approaches where you're storing a representation of data in one place, and then trying to work out the intersection of permissions from one of more other systems, and sanitise the results on the way out. Always seemed messy and fraught with problems and the risk of leaking something you shouldn't.

Re: The RAG Obituary: Killed by agents, buried by context windows

#89

Post by bjornsing »

davidmckayv wrote: post #42

This glosses over a fundamental scaling problem that undermines the entire argument. The author's main example is Claude Code searching through local codebases with grep and ripgrep, then extrapolates this to claim RAG is dead for all document retrieval. That's a massive logical leap. Grep works great when you have thousands of files on a local filesystem that you can scan in milliseconds. But most enterprise RAG use…

But couldn’t an LLM search for documents in that enterprise knowledge base just like humans do, using the same kind of queries and the same underlying search infrastructure?

Re: The RAG Obituary: Killed by agents, buried by context windows

#90

Post by elliotto »

cmenge wrote: post #64

We're processing tenders for the construction industry - this comes with a 'free' bucket sort from the start, namely that people practically always operate only on a single tender. Still, that single tender can be on the order of a billion tokens. Even if the LLM supported that insane context window, it's roughly 4GB that need to be moved and with current LLM prices, inference would be thousands of dollars. I detaile…

I work at an AI startup, and we've explored a solution where we preprocess documents to make a short summary of each document, then provide these summaries with a tool call instruction to the bot so it can decide which document is relevant. This seems to scale to a few hundred documents of 100k-1m tokens, but then we run into issues with context window size and rot. I've thought about extending this as a tree based structure, kind of like an LLM file system, but have other priorities at the moment.

Embeddings had some context size limitations in our case - we were looking at large technical manuals. Gemini was the first to have a 1m context window, but for some reason its embedding window is tiny. I suspect the embeddings might start to break down when there's too much information.

Post reply on HN