Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is RAG the Future of LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Ask HN: Is RAG the Future of LLMs?

#51

Post by spencerchubb »

waldrews wrote: post #46
gandalfgeek wrote: post #23

#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained. The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data…

We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That pricing model, and the context window limits, are a consequence of the quadratic cost of transformers though, and whatever the big context models like Gemini 1.5 are doing must have an (undisclosed) workaround. What needs to happen is a way to cheaply suspend and re…

When you describe the overlay layer, that sounds similar to the idea of low rank adaptation (LoRA). LoRA is kind of like finetuning, but it doesn't update every parameter, it adds a relatively small number of parameters and finetunes those

Am I understanding what you're describing about the VMs and containers analogy?

Re: Ask HN: Is RAG the Future of LLMs?

#52

Post by waldrews »

spencerchubb wrote: post #51
waldrews wrote: post #46

Earlier quoted context omitted.

We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That pricing model, and the context window limits, are a consequence of the quadratic cost of transformers though, and whatever the big context models like Gemini 1.5 are doing must have an (undisclosed) workaround. What needs to happen is a way to cheaply suspend and re…

When you describe the overlay layer, that sounds similar to the idea of low rank adaptation (LoRA). LoRA is kind of like finetuning, but it doesn't update every parameter, it adds a relatively small number of parameters and finetunes those Am I understanding what you're describing about the VMs and containers analogy?

Yup. I guess LoRA counts as fine tuning. Except I've never seen inference engines where they actually let you take the base model and the LoRA parameters as separate inputs (maybe it exists and I just haven't seen it). Instead, they bake the LoRA part into the bigger tensors as the final step of the fine tune. That makes sense in terms of making inference faster, but prevents the scenario where a host can just run the base model with any finetune you like, maybe switching them mid-conversation. Instead, if you want to host a fine-tuned model, you take the tensor blob and run a separate instance of the inference program on it. Incidentally, this is the one place where OpenAI and Azure pricing differs; OpenAI just charges you a big per-token premium for fine-tuned 3.5, and Azure charges you for the server to host the custom model. Likewise, the hosts for the open-weights models will charge you more to run your fine-tuned model than a standard model, even though it's the almost the same amount of GPU cycles, just because it needs to run on a separate server that won't be shared by multiple customers; that wouldn't be necessary if overlays were separated.

I wouldn't be surprised if GPT-4's rumored mixture of many models does something like this overlay management internally.

Re: Ask HN: Is RAG the Future of LLMs?

#53

Post by dlubarov »

p1esk wrote: post #49
simonw wrote: post #40

Earlier quoted context omitted.

A Large Language Model itself can't perform RAG: a model is a big binary blob of matrices that you run prompts against. Anything that can do RAG is, by definition, a system that wraps an LLM with additional code that performs the retrieval. It's the difference between ChatGPT (software that wraps a model and can extra features such as tool usage, Code Interpreter, RAG lookup via Bing etc) and GPT-4 Turbo (a model).

Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?

I think it's just a matter of semantics - "model" usually refers to a neural network or some similarly pure, deterministic computation, or so I thought.

Re: Ask HN: Is RAG the Future of LLMs?

#54

Post by simonw »

p1esk wrote: post #49
simonw wrote: post #40

Earlier quoted context omitted.

A Large Language Model itself can't perform RAG: a model is a big binary blob of matrices that you run prompts against. Anything that can do RAG is, by definition, a system that wraps an LLM with additional code that performs the retrieval. It's the difference between ChatGPT (software that wraps a model and can extra features such as tool usage, Code Interpreter, RAG lookup via Bing etc) and GPT-4 Turbo (a model).

Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?

I'm talking about Large Language Models - the architectures behind most of the current generative text AI boom.

In order to use tools they need to be run as part of a system that grants them access to tools, eg via the reAct pattern. https://til.simonwillison.net/llms/python-react-pattern

Re: Ask HN: Is RAG the Future of LLMs?

#55

Post by machinelearning »

Prosammer wrote: post #32
machinelearning wrote: post #25

Both RAG and infinite contexts in their current states are hacks. Both waste compute because you have to re-encode things as text each time and RAG needs a lot of heuristics + a separate embedding model. Instead, it makes a lot more sense to pre-compute KV for each document, then compute values for each query. Only surfacing values when the attention score is high enough. The challenge here is to encode global positi…

uh yeah it works out of the box, this is how most RAG systems are designed, just look at pgvector for example.

Nope that’s not how most rag systems work today. I looked at pgvector and couldn’t find anything similar.

Do you have a link? Or maybe you misunderstood what I was taking about

Re: Ask HN: Is RAG the Future of LLMs?

#57

Post by p1esk »

simonw wrote: post #54
p1esk wrote: post #49

Earlier quoted context omitted.

Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?

I'm talking about Large Language Models - the architectures behind most of the current generative text AI boom. In order to use tools they need to be run as part of a system that grants them access to tools, eg via the reAct pattern. https://til.simonwillison.net/llms/python-react-pattern

Are you talking about a simple script which executes model’s instructions? It can literally be as simple as connecting the model to my computer command line. I will tell it what I want done and give it my cc number or website creds.

Obviously I’m talking about next gen models, like gpt5/6.

Re: Ask HN: Is RAG the Future of LLMs?

#58

Post by sigmoid10 »

simonw wrote: post #16
sigmoid10 wrote: post #6

The latest research suggests that the best thing you can do is RAG + finetuning on your target domain. Both give roughly equal percentage gains, but they are independent (i.e. they accumulate if you do both). As context windows constantly grow and very recent architectures move more towards linear context complexity, we'll probably see current RAG mechanisms lose importance. I can totally imagine a future where if yo…

Where is the research that says that finetuning on your target domain gives a roughly equal percentage gain to RAG? I've not seen that.

See e.g. this paper from Microsoft: https://arxiv.org/abs/2401.08406

Re: Ask HN: Is RAG the Future of LLMs?

#59

Post by 0x008 »

2099miles wrote: post #36
gandalfgeek wrote: post #23

#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained. The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data…

Yeah I say cost is the biggest thing. Why doesn’t everyone just use GPT 4 for everything or Gemini ultra + RAG with all documents in the rag system with the best embedding model Among other things because it’s way too expensive and narrowing your scope cuts huge costs and isn’t hard to do at a high level

There is also the problem that most of the LLMs of today will somehow lose (or ignore) the middle if the context and prefer beginning and/or end.

Re: Ask HN: Is RAG the Future of LLMs?

#60

Post by 0x008 »

cl42 wrote: post #19

RAG will have a place in the LLM world, since it's a way to obtain data/facts/info for relevant queries. Since you asked about alternatives... (a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2]. (b) Depending on how you define RAG, knowledge graphs co…

I don't understand why knowledge graph would be an alternative to RAG? Knowledge graphs can (and are already) used as part of a RAG pipeline.
Post reply on HN