What will happen as inference cost goes down is RAG will just be one master LLM calling a bunch of smaller LLMs spanning the context window of every document you are querying. Context windows went from 8k to like 128k or more in like a year. In a few years we will have practically unlimited context windows for minimal cost.
Ask HN: Is RAG the Future of LLMs?
91–100 of 108 posts
Re: Ask HN: Is RAG the Future of LLMs?
#92Does RAG depend on a vector database?
tl;dr we only need a vector database if we want to do semantic vector-embedding search AND our dataset is too large for memory long answer: RAG is just a pattern of working with LLMs, independent of particular database technologies. Basically it means you inject some context data or domain specific data into the prompt to achieve the following: 1. Nudging the model into the right direction so it will use the "correct…
This is a great explanation for someone like me, ie total novice.
You’ve explained it in a way so that each part of the abbreviation makes sense now.
Thanks!
Re: Ask HN: Is RAG the Future of LLMs?
#93For those of us who don’t know what RAG is (including myself), RAG stands for Retrieval Augmented Generation. From the video in this IBM post [0], I understand that it is a way for the LLM to check what its source and latest date of information is. Based on that, it could, in principle, say “I don’t know”, instead of “hallucinating” an answer. A RAG is a way to implement this feature for LLMs. [0] https://research.ib…
The best way to understand RAG is that it's a prompting hack where you increase the chance that a model will answer a question correctly by pasting a bunch of text that might help into the prompt along with their question. The art of implementing RAG is deciding what text should be pasted into the prompt in order to get the best possible results. A popular way to implement RAG is using similarity search via vector se…
An additional (and I was quite surprised by this) trick is to ask an LLM to reformulate the user prompt “to be more concise and precise”, then run a vector similarity search against that, which (in our experience) leads to better matches.
There are many tricks to get better context to send to your LLM, and that’s a large part of making the system give good answers.
Re: Ask HN: Is RAG the Future of LLMs?
#94Unless we’re going to paste a whole domain corpus into the context window, we’re going to continue to need some sort of “relevance function” - a means of discriminating what needs to go in from what doesn’t. That could be as simple as “document A goes in, document B doesn’t”. That’s RAG. Doesn’t matter that you didn’t use vectors or knowledge graphs or FTS or what have you. Then the jump from “this whole document” to…
Recursive retrieval, although this is too early to say, is trying to solve for the smaller chunk sized documents. I still understand that the need for semantic map will persist.
Re: Ask HN: Is RAG the Future of LLMs?
#95Earlier quoted context omitted.
A Large Language Model itself can't perform RAG: a model is a big binary blob of matrices that you run prompts against. Anything that can do RAG is, by definition, a system that wraps an LLM with additional code that performs the retrieval. It's the difference between ChatGPT (software that wraps a model and can extra features such as tool usage, Code Interpreter, RAG lookup via Bing etc) and GPT-4 Turbo (a model).
Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?
Though the terminology is incorrect (a model by itself cannot 'perform' word other than encode/decode, the llm agents are developed to do what you're stating. But when you think about an llm agent doing all sorts of things like sql etc, it effectively becomes a RAG (which in itself is a process than a technology)
Re: Ask HN: Is RAG the Future of LLMs?
#96What I Observe: Simple RAG is Fading, but Complex RAG Will Persist and Evolve - Involving Query Rewriting, Data Cleaning, Reflection, Vector Search, Graph Search, Rerankers, and More Intelligent Chunking. Large Models Should Not Just Be Knowledge Providers, But Tool Users and Process Drivers"
It's becoming so complex that it will stop being called RAG. It's just an application that uses an LLM as one part of it.
I see many folks misunderstanding RAG for a technology. It's just a process. No matter the complexity, if the underlying principal is to augment the LLM with specific information to the question at hand - it is RAG.
Re: Ask HN: Is RAG the Future of LLMs?
#97Earlier quoted context omitted.
Why can’t a model explore its environment (given access) and find and use those tools? Why can’t a model fire up a query (sql, http, or whatever) to do the retrieval if it determines it needs more information?
Though the terminology is incorrect (a model by itself cannot 'perform' word other than encode/decode, the llm agents are developed to do what you're stating. But when you think about an llm agent doing all sorts of things like sql etc, it effectively becomes a RAG (which in itself is a process than a technology)
Does the LLM get trained as the database get builded?
As you train a model it should build it's knowledge database at the same time. I guess that's how it work?
When an LLM get trained on next word guessing, can it search the web to improve his results?
Re: Ask HN: Is RAG the Future of LLMs?
#98RAG can't create associations to data that isn't superficially (found by the indexing strategy) assosciated to the query. For example, you might query about one presidential candidate and lose out on the context of all other presidential candidates (probably a bad example, but gets the point across).
It is "search and summarize." It is not "glean new conclusions." That being said, "search and summarize" is probably good for 80%.
LoRA is an improvement, but I have seen benchmarks showing that it struggles to make as deep inferences as regular training does.
There isn't a one-size fits all... Yet.
Re: Ask HN: Is RAG the Future of LLMs?
#99RAG can't create associations to data that isn't superficially (found by the indexing strategy) assosciated to the query. For example, you might query about one presidential candidate and lose out on the context of all other presidential candidates (probably a bad example, but gets the point across). It is "search and summarize." It is not "glean new conclusions." That being said, "search and summarize" is probably g…
Well, theoretically you should be able to replace the "search" part of "search and summarize" with more analytics-y things - counts, aggregations, joins, whatever - and throw some prompt formatting at it and I'll bet you can get some pretty good conclusions out of an LLM.
Not sure you can call it RAG, but that can probably cover a good 90% of the remaining 20%
Re: Ask HN: Is RAG the Future of LLMs?
#100RAG is mostly a hack to address limited context windows, or limited use of wide context windows – some models have large windows but don’t use content that isn’t near the beginning or end well, or expensive content windows (LLM-as-service typically charges by the token, so RAG can reduce cost).
“Stuffing relevant data into the context window rather than relying purely on training” is a solution to confabulation, though, just like providing relevant reference information to a person who is being pressured to answer a question is.