Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is RAG the Future of LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 108 posts

Re: Ask HN: Is RAG the Future of LLMs?

#21

Post by fallingknife »

spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

That's going to be quite a while then because LLMs don't even perform well at anything close to the current context limit. They will miss important details when the context gets over a couple thousand tokens.

Re: Ask HN: Is RAG the Future of LLMs?

#22

Post by ttul »

spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

https://arxiv.org/html/2404.07143v1

“This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

Re: Ask HN: Is RAG the Future of LLMs?

#23

Post by gandalfgeek »

#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained.

The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data specific to the user issuing the query (their orders, history etc) and adding that to the context.

So will very large context windows (1M tokens!) "kill RAG"?

- at the simple end of the app complexity spectrum: when you're spinning up a prototype or your "corpus" is not very large, yes-- you can skip the complexity of RAG and just dump everything into the window.

- but there are always more complex use-cases that will want to shape the answer by limiting what they put into the context window.

- cost-- filling up a significant fraction of a 1M window is expensive, both in terms of money and latency. So at scale, you'll want to filter out and RAG relevant info rather than indiscriminately dump everything into the window.

Re: Ask HN: Is RAG the Future of LLMs?

#24

Post by verdverm »

simonw wrote: post #15
spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

It's not just about context length, it's about performance. A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things. As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context.

Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions

Re: Ask HN: Is RAG the Future of LLMs?

#25

Post by machinelearning »

Both RAG and infinite contexts in their current states are hacks.

Both waste compute because you have to re-encode things as text each time and RAG needs a lot of heuristics + a separate embedding model.

Instead, it makes a lot more sense to pre-compute KV for each document, then compute values for each query. Only surfacing values when the attention score is high enough.

The challenge here is to encode global position information in the surfaced values and to get them to work with generation. I suspect it can't be done out of the box but we it will work with training.

This approach has echoes of both infinite context length and RAG but is an intermediate method that can be parallelized and is more efficient than either one.

Re: Ask HN: Is RAG the Future of LLMs?

#26

Post by spencerchubb »

simonw wrote: post #15
spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

It's not just about context length, it's about performance. A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things. As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

Good point! If datasets get that large that is an argument in favor of RAG

Re: Ask HN: Is RAG the Future of LLMs?

#27

Post by grugagag »

cl42 wrote: post #19

RAG will have a place in the LLM world, since it's a way to obtain data/facts/info for relevant queries. Since you asked about alternatives... (a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2]. (b) Depending on how you define RAG, knowledge graphs co…

I hope it goes away because I have a real carpet to replace it with

Re: Ask HN: Is RAG the Future of LLMs?

#29

Post by haolez »

I don't think so. Token windows are always increasing and new architectures (Le Cunn is proposing some interesting stuff with world models) might make it cheaper to add knowledge to the model itself. I think it's more of a necessity of our current state of the art than something that I'd bet on.

Re: Ask HN: Is RAG the Future of LLMs?

#30

Post by p1esk »

It’s strange: most answers here assume the next gen models won’t be able to perform RAG on its own. IMO, it would be wise to assume the opposite - anything humans currently do to make models smarter will be built in.
Post reply on HN