Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is RAG the Future of LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 108 posts

Re: Ask HN: Is RAG the Future of LLMs?

#41

Post by thatjoeoverthr »

With this technology, faster chips will solve it _for an application-specific definition of "solved."_

The models aren't actually capable of taking into account everything in their context window with industrial yields.

These are stochastic processes, not in the "stochastic parrot" sense, but in the sense of "you are manufacturing emissions and have some measurable rate of success." Like a condom factory.

When you reduce the amount of information you inject, you both decrease cost and improve yield.

"RAG" is application specific methods of estimating which information to admit to the context window. In other words, we use domain knowledge and labor to reduce computational load.

When to do that is a matter of economy.

The economics of RAG in 2024 differ from 2022, and will differ in 2026.

So the question that matters is, "given my timeframe, and current pricing, do I need RAG to deliver my application?"

The second question is, "what's an acceptable yield, and how do I measure it?"

You can't answer that for 2026, because, frankly, you don't even know what you'll be working on.

Re: Ask HN: Is RAG the Future of LLMs?

#42

Post by nl »

In the ~2 year timeframe we'll be using RAG.

Longer term it gets more interesting.

Assuming we can solve long (approaching infinite) context, and solve the issues with reasoning over long context that LangChan correctly identified[1] then it becomes a cost and performance (speed) issue.

It is currently very very expensive to run a full scan of all knowledge for every inference call.

And there are good reasons why databases use indexes instead of table scans (ie, performance).

But maybe we find a route forward towards adaptive compute over the next two years. Then we can use low compute to find items of interest in the infinite contest window, and then use high compute to reason over them. Maybe this could provide a way forward on the cost issues at least.

Performance is going to remain an issue. It's not clear to me how solvable that is (sure you can imagine ways it could be parallelized but it seems likely there will be a cost penalty on planning that)

[1] https://blog.langchain.dev/multi-needle-in-a-haystack/

Re: Ask HN: Is RAG the Future of LLMs?

#43

Post by muratsu »

gandalfgeek wrote: post #23

#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained. The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data…

In addition to what you’ve shared, I find RAG to be useful for cases where LLM has the world knowledge (say it knows how to write javascript) but I want it to follow a certain style or dependencies (eg use function definitions vs function expressions, newest vs es6, etc). From what I’ve heard, it’s still cheaper/more performant to feed everything into the context than finetune models.

Re: Ask HN: Is RAG the Future of LLMs?

#44

Post by waldrews »

I work on statistical quality control methods for the hallucination problem. Model how difficult/error prone a query is, and prioritize sending it to humans to verify the LLM's answer if it's high risk. Some form of human control like that is the only way to really cut hallucinations down to something like human-equivalent level (human answers are unreliable too, and should be subject to quality control with reputation scores and incentives as well).

RAG can augment the LLM with specific knowledge, which may make it more likely to give factually correct answers in those domains, but is mostly orthogonal to the hallucination problem (except to the extent that LLM's hallucinate when asked questions on a subject they don't know).

Re: Ask HN: Is RAG the Future of LLMs?

#46

Post by waldrews »

gandalfgeek wrote: post #23

#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained. The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data…

We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That pricing model, and the context window limits, are a consequence of the quadratic cost of transformers though, and whatever the big context models like Gemini 1.5 are doing must have an (undisclosed) workaround.

What needs to happen is a way to cheaply suspend and rehydrate the memory state of the forward pass after you've fed it a lot of tokens.

That would be a sort of light-weight/flexible/easily modifiable/versionable/real-time-editable alternative to fine tuning.

It's readily doable with the open weights LLM's, but none of them (yet) have the context length to make it really worthwhile (some of the coding LLM's have long context windows, but it doesn't solve the 'knowledge base' scenario).

From a hosting perspective, if fine tunes are like VM's, such frozen overlays are like docker containers: many versions can live on the same server, sharing the base model and differing in the overlay layer.

(a startup idea? who wants to collaborate on a proof of concept?)

Re: Ask HN: Is RAG the Future of LLMs?

#49

Post by p1esk »

simonw wrote: post #40
p1esk wrote: post #30

It’s strange: most answers here assume the next gen models won’t be able to perform RAG on its own. IMO, it would be wise to assume the opposite - anything humans currently do to make models smarter will be built in.

A Large Language Model itself can't perform RAG: a model is a big binary blob of matrices that you run prompts against. Anything that can do RAG is, by definition, a system that wraps an LLM with additional code that performs the retrieval. It's the difference between ChatGPT (software that wraps a model and can extra features such as tool usage, Code Interpreter, RAG lookup via Bing etc) and GPT-4 Turbo (a model).

Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?

Re: Ask HN: Is RAG the Future of LLMs?

#50

Post by p1esk »

2099miles wrote: post #34
p1esk wrote: post #30

It’s strange: most answers here assume the next gen models won’t be able to perform RAG on its own. IMO, it would be wise to assume the opposite - anything humans currently do to make models smarter will be built in.

I mean you can’t build rag into a model, it’s part of a system or application. You wouldn’t say a steering wheel is built into an engine. People just build cars with engines.

I guess I’m a bit confused - what would be an equivalent to a “steering wheel” in the context of a human brain as an “engine”? Or would this steering wheel be part of the brain? If so, why do we need to separate the two in the context of an LLM?
Post reply on HN