Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From zero to a RAG system: successes and failures

en.andros.dev

Post reply on HN
Print view View original

121–129 of 129 posts

Re: From zero to a RAG system: successes and failures

#121

Post by mileycyrusXOXO »

KPGv2 wrote: post #44

This article came just in the nick of time. I'm in fandoms that lean heavily into fanfiction, and there's a LOT out there on Ao3. Ao3 has the worst search (and yo can't even search your account's history!), so I've been wanting to create something like this as a tool for the fandom, where we can query "what was the fic about XYZ where ABC happened?" and get hopefully helpful responses. I'm very tired of not being abl…

I did something similar to this for all the Cosmere stuff. I wanted to be able to find answers but only with the information I had read been exposed to so far. I didn't want to risk going to the wiki and getting spoilers for things I haven't read yet. It wasn't anything fancy, it was just giving the agent access to all the text I had read up to my current chapter. Probably too much context for it to handle efficiently - would be awesome to take it one step further and do it proper

Re: From zero to a RAG system: successes and failures

#122

Post by joefourier »

hbrn wrote: post #120
joefourier wrote: post #77

Earlier quoted context omitted.

I agree with you that simple vector search + context stuffing is dead as a method, but I think it's ridiculous to reserve the term "RAG" for just the earliest most basic implementation. The definition of Retrieval Augmented Generation is any method that tries to give the LLM relevant data dynamically as opposed to relying purely on it memorising training data, or giving it everything it could possibly need and relyin…

> it's ridiculous to reserve the term "RAG" for just the earliest most basic implementation Whether we like it or not, dumb semantic search became the colloquial definition of RAG. And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean. When you inject user's name into the system prompt, technically you're doing RAG - but nobody thinks about it that way. I think it's one of…

Then what do you call RAG done well? You need a term for it.

> And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean.

That's just Sturgeon's law in action. 95% of every implementation is crap. Back in the 90s, you might have heard "we use OOP here" and come to a similar conclusion, but that doesn't mean you need to invent a new word for doing OOP properly.

> But agentic RAG is fundamentally different.

From an implementation POV, absolutely not.

I've personally gradually converted a dumb semantic search to a more fully featured agentic RAG in small steps like these:

- Have a separate LLM call write the query instead of just using the user's message. - Make the RAG search a synthetic injected tool call, instead of appending it to the system prompt. - Improve the search endpoint by using an LLM to pre-process the data into structured chunks with hierarchical categories, tags, and possible search queries, embedding the search queries separately from the desired information (versus originally just having a raw blob). - Have the LLM be able to search both with a semantic sentence, and a list of tags. - Have the LLM view and navigate the hierarchy in a tree-like manner. - Make the original LLM able to call the search on its own instead of being automatically injected using a separate query rewriting call, letting it search in multiple rounds and refine its own queries.

When did the system go from RAG to "not RAG"? Because fundamentally, all you need to do to make an agentic RAG is to have the LLM be able to write/rewrite its own search queries (possibly in multiple passes) as opposed to just passing the user's messages(s) directly.

Re: From zero to a RAG system: successes and failures

#123

Post by tren_hard »

notglossy wrote: post #107
tren_hard wrote: post #79

Earlier quoted context omitted.

I’m still learning this advantages and differences between them, would there be benefits to SFT and RAG? Or does RAG make SFT redundant?

I think generally, SFT is like giving the LLM increased intuition in specific areas. If you combine this with RAG, it should improve the performance or accuracy. Sort of like being a lawyer and knowing something is against the law by intuition, but needing the library to cite a specific case or statute as to why.

Thank you I appreciate the reply and that analogy helps make sense of this.

Re: From zero to a RAG system: successes and failures

#124

Post by hbrn »

joefourier wrote: post #122
hbrn wrote: post #120

Earlier quoted context omitted.

> it's ridiculous to reserve the term "RAG" for just the earliest most basic implementation Whether we like it or not, dumb semantic search became the colloquial definition of RAG. And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean. When you inject user's name into the system prompt, technically you're doing RAG - but nobody thinks about it that way. I think it's one of…

Then what do you call RAG done well? You need a term for it. > And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean. That's just Sturgeon's law in action. 95% of every implementation is crap. Back in the 90s, you might have heard "we use OOP here" and come to a similar conclusion, but that doesn't mean you need to invent a new word for doing OOP properly. > But agentic RAG…

>all you need to do to make an agentic RAG is to have the LLM be able to write/rewrite its own search queries (possibly in multiple passes)

I think this is a huge oversimplification, the term "search query" is doing a lot of heavy lifting here.

When Claude Code calls something like

  find . -type d -maxdepth 3 -not -path '*/node_modules/*'
to understand the project hierarchy before doing any of the grep calls, I don't think it's fair to call it just a "search query", it's more like "analyze query". Just because text goes in and out in both cases, doesn't mean that it's all the same.

When you give the agent the ability to query the nature of the data (e.g. hierarchy), and not just data itself, it means that you need to design your product around it. Agentic RAG has entirely different implementation, product implications, cost, latency, and primarily, outcomes. I don't think it's useful to pretend that it's just a different flavor of the same thing, simply because at the end of the day it's just some text flying over the network.

Re: From zero to a RAG system: successes and failures

#125

Post by ozim »

joefourier wrote: post #122
hbrn wrote: post #120

Earlier quoted context omitted.

> it's ridiculous to reserve the term "RAG" for just the earliest most basic implementation Whether we like it or not, dumb semantic search became the colloquial definition of RAG. And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean. When you inject user's name into the system prompt, technically you're doing RAG - but nobody thinks about it that way. I think it's one of…

Then what do you call RAG done well? You need a term for it. > And when you hear someone saying "we use RAG here" 95% of the time this is exactly what they mean. That's just Sturgeon's law in action. 95% of every implementation is crap. Back in the 90s, you might have heard "we use OOP here" and come to a similar conclusion, but that doesn't mean you need to invent a new word for doing OOP properly. > But agentic RAG…

I like the audacity of parent poster that equates 95% of implementations he has seen with 95% of all there is. When it easily could have been 0.01% of all there is. World is much bigger than we think :)

Re: From zero to a RAG system: successes and failures

#126

Post by ozim »

So 95% of the post is „regular software engineering” like, yes you cannot just process 1TB of data, you need to split it up then even if you split it up you might have limited budget for processing so think how you fit in that, make checkpoints and make sure you have logs.

Not dismissing the value of the blog post. Just underlining for „non engineers”.

Re: From zero to a RAG system: successes and failures

#127

Post by thisischayan »

notglossy wrote: post #12
mentos wrote: post #11

Earlier quoted context omitted.

I assume it’s not possible to get the same results by fine tuning a model with the documents instead?

You will still get hallucinations. With RAG you use the vectors to aid in finding things that are relevant, and then you typically also have the raw text data stored as well. This allows you to theoretically have LLM outputs grounded in the truth of the documents. Depending on implementation, you can also make the LLM cite the sources (filename, chunk, etc).

The approach that has worked for us in production is correction during generation, not after.

The model verifies its output against the rules in the prompt as it generates and corrects itself within the same API call — no retries, no external validator. If there are still failures the model cannot fix at runtime, those are explicitly flagged instead of silently producing wrong output.

This does not mean hallucinations are completely solved. It turns them into a measurable engineering problem. You know your error rate, you know which outputs failed, and you can drive that rate down over time with better rules. The system can also self-learn and self-improve over time to deliver better accuracy.

Re: From zero to a RAG system: successes and failures

#128

Post by martinarva »

_the_inflator wrote: post #66

I implemented many RAGs and feel sorry for anyone proclaiming "RAG is dead". These folks have never implemented one, maybe followed a tutorial and installed a "Hello World!" project but that's it. I don't want to go into detail but 100% agree with the author's conclusion: data is key. Data ingestion to be precisely. Simply using docling and transforming PDFs to markdown and have a vector database doing the rest is ri…

[dead]
Post reply on HN