61–70 of 108 posts
With RAG tools that exist today, we can already do things like
- providing summaries
- hierarchical summarization
- generation of questions / more prompts to nudge the model
- caching
- using knowledge graphs, function calling, or database connectors for non-semantic data querying
etc.
I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimiz…
I think that "next level" would essentially be a RAG referential information system that gets data via search engines or databases. Maybe we will have the "google search" equivalent completely intended for LLM clients where all data is stored, searched, and returned via vector embeddings, but it could tap on exabytes of information.
Earlier quoted context omitted.
I'm talking about Large Language Models - the architectures behind most of the current generative text AI boom. In order to use tools they need to be run as part of a system that grants them access to tools, eg via the reAct pattern. https://til.simonwillison.net/llms/python-react-pattern
Are you talking about a simple script which executes model’s instructions? It can literally be as simple as connecting the model to my computer command line. I will tell it what I want done and give it my cc number or website creds. Obviously I’m talking about next gen models, like gpt5/6.
In the ~2 year timeframe we'll be using RAG. Longer term it gets more interesting. Assuming we can solve long (approaching infinite) context, and solve the issues with reasoning over long context that LangChan correctly identified[1] then it becomes a cost and performance (speed) issue. It is currently very very expensive to run a full scan of all knowledge for every inference call. And there are good reasons why dat…
Instead of computing every token every time on the whole context, we can grab a cache to make some shortcut. We do the same in software development all the time. Of course it's a performance issue.
I don't think so. Token windows are always increasing and new architectures (Le Cunn is proposing some interesting stuff with world models) might make it cheaper to add knowledge to the model itself. I think it's more of a necessity of our current state of the art than something that I'd bet on.
We cannot simply state that at some point in time RAG will not be necessary. Like everything in the computer science world it always will depend on our data size and the resource constraints we have.
Unless of course we can process a corpus the size of the whole internet in <1 second. However, I doubt this can be achieved in the next 20 years.
Does RAG depend on a vector database?
we only need a vector database if we want to do semantic vector-embedding search AND our dataset is too large for memory
long answer:
RAG is just a pattern of working with LLMs, independent of particular database technologies.
Basically it means you inject some context data or domain specific data into the prompt to achieve the following:
1. Nudging the model into the right direction so it will use the "correct parts" of it's "global knowledge". We want to nudge it into the correct domain or a domain very similar to our problem. We do this to generate better answers and reduce hallucinations or to adhere to a particular style we want to achieve. This is basically just a prompt-engineering technique.
2. Achieving in-context learning by providing some context to the model. This is also a prompt engineering technique used to make the model reason about things it does not know from the training and to allow it to reason about a particular text you provide at runtime.
What I just explained is basically the "AG"-Part of RAG: Augmented Generation. It boils down to putting some external data into the context.
The R (Retrieval), on the other hand, is about finding the correct data to put in the context.
If you copy an email and paste into the prompt of ChatGPT, you are basically already doing RAG. The R is copy pasting the Email from Gmail, and the AG is putting it into the Prompt.
So, no, generally speaking you don't need a vector Database.
In the real world you will probably have a large context (some files on the disk, a PDF file, etc.) and you don't want to manually select the relevant bits of information, of course. We want to automate the Retrieval part of RAG. This is mostly what people mean when they talk about "RAG". Essentially it is a information retrieval, or search problem.
A lot of programmers and projects which come from the LLM community will use RAG with a semantic vector search. Semantic vector search has shown to be quite good at selecting relevant context chunks from full text and it also feels natural because it will use embedding models, which are very similar to LLMs. By using vector-embedding search we simulate with a manual step what we think will happen when we put all the data into the LLM context: Selection through semantic interpretation.
In this case we need some form of "vector search", but we do not need a database for that per se, we can also do that in-memory. However, if our dataset is too large to be efficiently processed in-memory, we would need a vector database.
However, we can also use any other means of information-retrieval techniques or search techniques, like keyword search, full-text search, knowledge graphs, sql queries, and many more, to retrieve the relevant bits of information we want to pass to the model.
In that case we also don't need a vector database. Maybe some other database, maybe a search tool like elastic.io, maybe no database at all.
#1 motivation for RAG: you want to use the LLM to provide answers about a specific domain. You want to not depend on the LLM's "world knowledge" (what was in its training data), either because your domain knowledge is in a private corpus, or because your domain's knowledge has shifted since the LLM was trained. The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data…
We're getting large context windows, but so long as pricing is by the input token, the 'throw everything into the context window' path isn't viable. That pricing model, and the context window limits, are a consequence of the quadratic cost of transformers though, and whatever the big context models like Gemini 1.5 are doing must have an (undisclosed) workaround. What needs to happen is a way to cheaply suspend and re…
But frontier models are just too damn good and convenient so I don't think its possible to fully get away from web LLMs.
i am sure there can be newer ways to do prompt injection in an elegant way, but for the most part the llm is either summarizing the injected prompt or regurgitating it.
if the output is satisfactory, it is still more convenient than writing custom rules for answers for each kind of question you want to address.
The best solution to reducing the problem of Hallucinations is first someone telling us what their Error rates in production are.
(In response to “The best solution to reducing the problem of Hallucinations is first someone telling us what their Error rates in production are.”)
If I were to make an educated guess, one response would be akin to “one cannot correct errors without measuring the errors at prediction time”. This is incorrect; measuring prediction errors is not the only way.
Some kinds of errors can be reduced or even eliminated before they are generated. Consider the space of models that are structurally incapable of making certain classes of errors.
To explain it in another way: “turning the crank on a model” isn’t the only way to reason about it.
For example, in statistical modeling, there are kinds of models that do not suffer from out of sample extrapolation.
Also, the ideas of trust regions (from RL) might inspire alternative methods worth exploring.