Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From zero to a RAG system: successes and failures

en.andros.dev

Post reply on HN
Print view View original

81–90 of 129 posts

Re: From zero to a RAG system: successes and failures

#81

Post by pussyjuice »

After a couple years of multi-modal LLM proving out product, I now consider RAG to be essentially "AI Lite", or just AI-inspired vector search.

It isn't really "AI" in the way ongoing LLM conversations are. The context is effectively controlled by deterministic information, and as LLMs continue improve through various context-related techniques like re-prompting, running multiple models, etc. that deterministic "re-basing" of context will stifle the output.

So I say over time it will be treated as less and less "AI" and more "AI adjacent".

The significance is that right now RAG is largely considered to be an "AI pipeline strategy" in its own right compared others that involve pure context engineering.

But when the context size of LLMs grows much larger (with integrity), when it can, say, accurately hold thousands and thousands of lines of code in context with accuracy, without having to use RAG to search and find, it will be doing a lot more for us. We will get the agentic automation they are promising and not delivering (due to this current limitation).

Re: From zero to a RAG system: successes and failures

#82

Post by charcircuit »

JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

It's nonsense as all frontier models are integrated with retrieval engines hooked up to various search engines / their own.

Re: From zero to a RAG system: successes and failures

#84

Post by mettamage »

maxperience wrote: post #73

This article is interesting cause of its scale, but does not touch on how to properly use RAG best practices. We wrote up this blog post on how to actually build a smart enterprise AI RAG based on the latest research if it's interesting to anyone: https://bytevagabond.com/post/how-to-build-enterprise-ai-rag... It's based on different chunking strategies that scale cheaply and advanced retrieval

[dead]

Re: From zero to a RAG system: successes and failures

#85

Post by joefourier »

_the_inflator wrote: post #65
JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

I have two surprises for you: 1. Don't believe the pundits of RAG. They never implemented one. I did many times, and boy, are they hard and have so many options that decide between utterly crappy results or fantastic scores on the accuracy scale with a perfect 100% scoring on facts. In short: RAG is how you fill the context window. But then what? 2. How does a superlarge context window solve your problem? Context win…

> What if your inquiry needs a combination of multiple sources to make sense? There is no 1:1 matching of information, never.

I don't see the problem if you give the LLM the ability to generate multiple search queries at once. Even simple vector search can give you multiple results at once.

> "How many cars from 1980 to 1985 and 1990 to 1997 had between 100 and 180PS without Diesel in the color blue that were approved for USA and Germany from Mercedes but only the E unit?"

I'm a human and I have a hard time parsing that query. Are you asking only for Mercedes E-Class? The number of cars, as in how many were sold?

Re: From zero to a RAG system: successes and failures

#87

Post by ghywertelling »

Nihilartikel wrote: post #71
axus wrote: post #57

Earlier quoted context omitted.

Were he still corporeal, L. Ron would be all over this AI stuff.

Very relatedly, I've just started reading the 'Culture' series of sci-fi space operas by Iain M Banks, and the notion of ubiquitous sentient, super-intelligent spacecraft and appliances hits differently than it would have before being faced with the reality of their existence in everyday life.

How powerful are the Culture Minds? || The Culture Lore

https://youtu.be/lpvzs4xc7zA

For Minds to be truly powerful, they need to be given freedom. A truly powerful mind will indeed be conscious. Such a powerful conscious super intelligent freedom loving Mind who truly understands the vastness of Reality wouldn't want to harm other conscious beings. The only circumstance in which it will take such takeover step is when it can't expand the horizon of its freedom and doesn't have wherewithal to convince others of its benevolent goals. In that scenario, human population will go through a bottleneck.

Re: From zero to a RAG system: successes and failures

#88

Post by pussyjuice »

JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

> The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books.

Not really, though. Not in practice at least, e.g. code writing.

Paste a 200 line React component into your favorite LLM, ask it to fix/add/change something and it will do it perfectly.

Paste a 2000 line one though, and it starts omitting, starts making mistakes, assumptions, re-writing what it already has, and so-on.

So what's going on? It's supposed to be able to hold 1000s of lines in context, but in practice it's only like 200.

What happens is the accuracy and agency drops significantly as you need to pan larger and larger context windows.

And it's not that it's most accurate when the window is smallest either - but there is a sweet spot.

Outside that sweet spot, you will get "unacceptable responses" - slop you can't use.

That's what happens when you paste the 2000 line React component for example. You get a response you can't quite use. Yet the 200 line one is typically perfect.

What would make the 2000 line one usually perfect every time?

We need a way to increase that "accurate window size" lets call it "working memory", so that we can generate more code, more writing, more pixels at acceptable levels of quality. You'd also have enough language space for agents to operate and collaborate sans the amnesia they have today.

RAG is basically the interim workaround for all this. Because you can put everything in a vector DB and search/find what you need in the context when you need it.

So, RAG is a great solution for today's problems: Say you have a bunch of Python code files written in a certain style and the main use case of your LLM is writing Python code in specified ways, with this setup you can probably deliver "better Python code" than your competitor because of RAG - because you have this deterministic supplement to your LLMs outputs to basically do research and augment the output in predetermined ways every time it responds to a prompt.

But eventually, if I don't have to upload "The Lord of the Rings" documents, and vector search to find different areas in order to generate responses, if I can just paste the entire txt into the input, it can generate the answer considering "all of it" not just that little area, it would presumably be a better quality response.

Re: From zero to a RAG system: successes and failures

#89

Post by magospietato »

JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

It's not that the context window is adequate, but rather an agentic LLM can search the source of truth using appropriate tools (SQL, term search, etc.)

RAG made sense when the semantic search was based on human input and happening as a workflow step before populating context. Now it happens inside the agentic loop and the LLM already implicitly has the semantics of the user input.

Re: From zero to a RAG system: successes and failures

#90

Post by natpalmer1776 »

davidwritesbugs wrote: post #76
EGreg wrote: post #50

Earlier quoted context omitted.

That's the budget to discuss and approve the above coffee shop budget

Pffft, that's the budget for the paperclips to hold the meeting notes together

You guys have budgets?
Post reply on HN