Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is RAG the Future of LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 108 posts

Re: Ask HN: Is RAG the Future of LLMs?

#11

Post by epistasis »

mikecaulley wrote: post #8
spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

This doesn’t consider compute cost; the RAG model is much more efficient compared to infinite context length.

Agreed. I think that RAG implemented via tool-calling, with multiple agents talking to each other, is a much much more likely evolution in the future versus a single unified model.

I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.

Re: Ask HN: Is RAG the Future of LLMs?

#12

Post by darkteflon »

Unless we’re going to paste a whole domain corpus into the context window, we’re going to continue to need some sort of “relevance function” - a means of discriminating what needs to go in from what doesn’t. That could be as simple as “document A goes in, document B doesn’t”.

That’s RAG. Doesn’t matter that you didn’t use vectors or knowledge graphs or FTS or what have you.

Then the jump from “this whole document” to “well actually I only need this particular bit” puts you immediately into the territory of needing some sort of semantic map of the document.

I don’t think it makes conceptual sense to think about using LLMs without some sort of domain relevance function.

Re: Ask HN: Is RAG the Future of LLMs?

#13

Post by choilive »

tslmy wrote: post #9

To make a LLM relevant to you, your intuition might be to fine-tune it with your data, but: 1. Training a LLM is expensive. 2. Due to the cost to train, it’s hard to update a LLM with latest information. 3. Observability is lacking. When you ask a LLM a question, it’s not obvious how the LLM arrived at its answer. There’s a different approach: Retrieval-Augmented Generation (RAG). Instead of asking LLM to generate an…

This is the state of LLMs today - it is likely that we will have models in the future that can do some form of "online" training - or new training methods that aren't nearly as compute intensive. There are many people working on these scaling issues with LLMs today. We already have new attention heads that work around the quadratic time and space complexity of the input prompts.

Re: Ask HN: Is RAG the Future of LLMs?

#14

Post by nimish »

RAG is an easy way to incorporate domain knowledge into a generalized model.

It's 1000x more efficient to give it a look-aside buffer of info than to try to teach it ab initio.

Why do more work when the data is already there?

Re: Ask HN: Is RAG the Future of LLMs?

#15

Post by simonw »

spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

It's not just about context length, it's about performance.

A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.

As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

Re: Ask HN: Is RAG the Future of LLMs?

#16

Post by simonw »

sigmoid10 wrote: post #6

The latest research suggests that the best thing you can do is RAG + finetuning on your target domain. Both give roughly equal percentage gains, but they are independent (i.e. they accumulate if you do both). As context windows constantly grow and very recent architectures move more towards linear context complexity, we'll probably see current RAG mechanisms lose importance. I can totally imagine a future where if yo…

Where is the research that says that finetuning on your target domain gives a roughly equal percentage gain to RAG? I've not seen that.

Re: Ask HN: Is RAG the Future of LLMs?

#19

Post by cl42 »

RAG will have a place in the LLM world, since it's a way to obtain data/facts/info for relevant queries.

Since you asked about alternatives...

(a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2].

(b) Depending on how you define RAG, knowledge graphs could be a form of RAG or alternatively an alternative to them. Companies like Elemental Cognition[3] are building distinct alternatives to RAG that use such graphs and give LLMs the ability to run queries on said graphs. Another approach here is to build "fact databases" where, you structure observations about the world into standalone concepts/ideas/observations and reference those[4]. Again, similar to RAG but not quite RAG as we know it today.

[1] https://deepmind.google/discover/blog/alphageometry-an-olymp...

[2] https://arxiv.org/abs/2306.12672

[3] https://ec.ai/

[4] https://emergingtrajectories.com/

Re: Ask HN: Is RAG the Future of LLMs?

#20

Post by darby_eight »

spencerchubb wrote: post #5

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…

> I believe RAG is a temporary hack until we figure out virtually infinite context.

I'd assume "large enough" context is the actual goal here, not "virtually infinite".

Post reply on HN