RAG Is Simpler Than You Think

We removed retrieval from our coding agent a while back. What convinced us wasn’t a benchmark, we found that the retrieval path had been returning zero results for quite some time because of a technical bug, still nobody noticed, indeed it was working better than before.

After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.

Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.

(I work on TheGitAI, for disclosure.)

For people familiar with the field, it would be like if you had every article about hardware read “Intel Central Processing Units (CPUs) with modern Double Data Rate 5 (DDR5) Random Access Memory (RAM) can be coupled with Nvidia Graphics Processing Units (GPUs) to run Large Language Models (LLMs) that are stored on Solid State Disks (SSDs)”. Just rapidly becomes unreadable.

The acronym constraint was valid in a pre-LLM world but now you are perhaps 3 clicks in a modern browser from learning.

If I read an article that spelled out Random Access Memory I would definitely treat that as a lay article.

RAG is art. I have a very straight forward setup that is highly modular.

RAG is routing and decision making.

I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.

I agree with you.

Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.

It maintains itself.

Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.

I found my niche with RAG selling and I build them myself.

I take pride in them.

So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.

And there are regulated businesses where certain information is required to be always factual correct - pricing for example.

Vector search becomes a liability for this use case.

So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.

I love this kind of stuff.

And there is personal information etc.

Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.

And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.

It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.

I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.

RAG gives me dbCORE vibes so to say.

Good luck and fun with your RAG systems.

As simple as a prompting it with structural output or restrictions for your criteria.

With agents, the prompting could be dynamic for maximum accuracy for every retrieval.

This absolutely would beat the best of the best embedding-based RAG models.

Nobody uses this now mainly due to speed. An llm retrieval would be 10x or more slower than embedding.

You can try that now

Take some failing cases or bad retrieval from your current system Prompt an llm wisely like a perfect prompt to get what you want and provide it the context to it. And see the results.

For context, you are limited now by models contexts (1m), so mostly you would need to split what you have and prompt twice....or more...and so on

what I implied is that in 2024 a lot was written about RAG, LLMs were trained on it, and that explains your gpt-4o-mini reference. the argument is that the article is ai slop, and that point is totally unrelated to name dropping fortune 500s references