Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAG Obituary: Killed by agents, buried by context windows

nicolasbustamante.com

Post reply on HN
Print view View original

31–40 of 185 posts

Re: The RAG Obituary: Killed by agents, buried by context windows

#31

Post by nbstme »

djoldman wrote: post #16

... for this specific use case (financial documents). These corpora have a high degree of semantic ambiguity among other tricky and difficult to alleviate issues. Other types of text are far more amenable to RAG and some are large enough that RAG will probably be the best approach for a good while. For example: maintenance manuals and regulation compendiums.

Why? What if LLMs could parallelize much of their reading and then summarize the findings into a markdown file, eliminating the need for complicated search?

Re: The RAG Obituary: Killed by agents, buried by context windows

#32

Post by selcuka »

I don't find this surprising. We are constantly finding workarounds for technical limitations, then ditch them when the limitation no longer exists. We will probably be saying the same thing for LLMs in a few years (when a new machine learning related TLA becomes the hype).

Re: The RAG Obituary: Killed by agents, buried by context windows

#33

Post by nbstme »

aussieguy1234 wrote: post #25

grep was invented at a time when computers had very small amounts of memory, so small that you might not even be able to load a full text file. So you had tools that would edit one line at a time, or search through a text file one line at a time. LLMs have a similar issue with their context windows. Go back to GPT-2 and you wouldn't have been able to load a text file into its memory. Slowly the memory is increasing,…

Agree. It's a context/memory issue. Soon LLMs will have a 10M context window and they won't need to search. Most codebases are less than 10M tokens.

Re: The RAG Obituary: Killed by agents, buried by context windows

#34

Post by Analemma_ »

nbstme wrote: post #28
themanmaran wrote: post #6

I'm always amazed at claude codes ability to build context by just putting grep in a for loop. It's pretty much the same process I would use in an unfamiliar code base. Just ctrl+f the file system till I find the right starting point.

It's mind blowing. It's so simple, elegant and... effective! Grep+glob and a lot of iterations is all we need.

We always suspected find+grep+xargs was Turing-complete, and now Claude is proving it.

Re: The RAG Obituary: Killed by agents, buried by context windows

#35

Post by serf »

tptacek wrote: post #22
dymk wrote: post #14

Earlier quoted context omitted.

I don’t mind articles that have a hint of “an AI helped write this” as long as the content is actually informationally dense and well explained. But this article is an obvious ad, has almost no interesting information or summaries or insights, and has the… weirdly chipper? tone that AI loves to glaze readers with.

How is this an ad? It's a couple thousand words about how they built something complicated that was then obsoleted.

in the same vein that a 'Behind The Scenes Look At The Making of Jurassic Park' is , in fact, an ad.

having a company name pitched at you within the first two sentences is a pretty good give away.

Re: The RAG Obituary: Killed by agents, buried by context windows

#36

Post by hluska »

nbstme wrote: post #30
redwood wrote: post #18

Weird to see the use case referenced specifically code search when that's a very targeted one rather than what general purpose agents (or RAG) use cases might target.

The main use case I referenced is SEC filings search, which is quite different from code. Filings are much longer, less structured, and more complex, with tables and footnotes.

I’m sure that was your intent but why did you get bogged down talking about code?

Re: The RAG Obituary: Killed by agents, buried by context windows

#37

Post by nbstme »

selcuka wrote: post #32

I don't find this surprising. We are constantly finding workarounds for technical limitations, then ditch them when the limitation no longer exists. We will probably be saying the same thing for LLMs in a few years (when a new machine learning related TLA becomes the hype).

100%. The speed of change is wild. With each new model, we end up deleting thousands of lines of code (old scaffolding we built to patch the models’ failures.)

Re: The RAG Obituary: Killed by agents, buried by context windows

#38

Post by serf »

bigwheels wrote: post #10
sebmellen wrote: post #8

Earlier quoted context omitted.

Those guidelines that you reference talk almost exclusively about annoyances on the webpage itself, not the content of the article. I think it's fair to point out that many articles today are essentially a little bit of a human wrapper around a core of ChatGPT content. Whether or not this was AI-generated, the tells of AI-written text are all throughout it. There are some people who have learned to write like the AI…

Do you agree that bickering over AI-generated vs. not AI-generated makes for dull discussion? Sliding sewing needles deep into my fingernail bed sounds more appealing than nagging over such minutiae.

minutiae to me is the effort of loading a page and reading half a paragraph in order to determine the AI tone for myself. The new AI literature frontier has actually added value to reading the comments first on HN in a surprising twist -- saves me the trouble.

Re: The RAG Obituary: Killed by agents, buried by context windows

#39

Post by nbstme »

hluska wrote: post #36
nbstme wrote: post #30

Earlier quoted context omitted.

The main use case I referenced is SEC filings search, which is quite different from code. Filings are much longer, less structured, and more complex, with tables and footnotes.

I’m sure that was your intent but why did you get bogged down talking about code?

hum because Claude Code pioneered the 'grep/glob/read' paradigm, so I felt the need to explain that what works well for coding files can also be applied to more complex documents.

Re: The RAG Obituary: Killed by agents, buried by context windows

#40

Post by sergiotapia »

>The winners will not be the ones who maintain the biggest vector databases, but the ones who design the smartest agents to traverse abundant context and connect meaning across documents.

So if one were building say a memory system for an AI chat bot, how would you save all the data related to a user? Mother's name, favorite meals, allergies? If not a Vector database like pinecone, then what? Just a big .txt file per user?

Post reply on HN