Infinite Context LLMs: Going Beyond RAG with Extended Minds
blog.normalcomputing.ai
Infinite Context LLMs: Going Beyond RAG with Extended Minds
1–10 of 43 posts
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#2Oh interesting, from what I understand then, this is great for small context size models, compared to RAG? Is there research into how to make this more effective for large context size models (since context size of major models seems to be 4xing every 6 months at this point)?
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#3With almost all of these papers, "RAG" is mentioned as a well-defined, absolute strategy. Is there an agreed upon implementation for it? Because as I have been building my own implementation, I have found that finding the right things to retrieve and augment the prompt with is incredibly challenging.
It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the student a note with the answer right before the exam, compared to letting them read the whole curriculum, but I am very much still learning..
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#4Oh interesting, from what I understand then, this is great for small context size models, compared to RAG? Is there research into how to make this more effective for large context size models (since context size of major models seems to be 4xing every 6 months at this point)?
It appears that RAG actually dominates for 2k context lengths compared to this method, but that this method outperforms it more and more the longer the context gets (see the graph titled "Retrieval Benchmark Results, by Document Length")
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#5With almost all of these papers, "RAG" is mentioned as a well-defined, absolute strategy. Is there an agreed upon implementation for it? Because as I have been building my own implementation, I have found that finding the right things to retrieve and augment the prompt with is incredibly challenging. It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the stu…
I think it's still a developing technique. I'm not following RAG super closely, but I keep myself somewhat updated through Sam Witteveen's videos on YouTube: https://www.youtube.com/@samwitteveenai
His channel might be a useful resource for you.
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#6Oh interesting, from what I understand then, this is great for small context size models, compared to RAG? Is there research into how to make this more effective for large context size models (since context size of major models seems to be 4xing every 6 months at this point)?
It appears that RAG actually dominates for 2k context lengths compared to this method, but that this method outperforms it more and more the longer the context gets (see the graph titled "Retrieval Benchmark Results, by Document Length")
"Document length" is the length of the text that contains the answer. "Context length" is how much text the model can process to produce the answer, and this number is fixed across their experiments.
When the document length is 2k, it's likely smaller than the context and RAG can just retrieve the entire document to have the model read it. When the document is longer, RAG needs to actually do some work to pick the parts that contain the answer.
The "extended mind" can always query tokens across the entire document, though evidently worse than if they were included in the context.
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#7With almost all of these papers, "RAG" is mentioned as a well-defined, absolute strategy. Is there an agreed upon implementation for it? Because as I have been building my own implementation, I have found that finding the right things to retrieve and augment the prompt with is incredibly challenging. It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the stu…
I think it's still a developing technique. I'm not following RAG super closely, but I keep myself somewhat updated through Sam Witteveen's videos on YouTube: https://www.youtube.com/@samwitteveenai His channel might be a useful resource for you.
On the note of youtube channels, I can wholeheartedly recommend Prompt Engineering [1]. He has great videos and hands-on tutorials, not just about RAG, and also maintains the LocalGPT repo [2] with 17k stars.
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#8The blog post format is so much nicer than a PDF paper!
More seriously, this does feel like a real advance. Vector search+context stuffing (RAG) is clearly a hack that doesn't resemble how we actually think or do things in reality. I've been wondering for the last year whether it's possible to extend the attention mechanism to more naturally connect to a bigger set of per-session weights or activations. The moment you encounter the query/key/value analogy it's an obvious idea, the problem being that you need a very strong grip on the low level details of neural architecture to actually do it. Now apparently it is possible! And the way the topk knob actually maps to abstraction is quite amazing.
Still, this doesn't eliminate context window constraints. The memories themselves have a form of context window in this technique. Context size (what they call sequence length) does still matter.
Additionally, the memories have to actually fit in GPU memory (at least in their implementation). And the memories appear to be nearly full snapshots of the network, so they will get quite large (much larger than text grabbed using RAG). So there's going to be a painful tradeoff here for the forseeable future where you'll have to decide whether you want a bigger smarter base model with a bigger context window but less space for memory, or a smaller model with a smaller context window but bigger memory.
This is the first I've heard of Normal Computing, who are these guys/gals exactly?
> Normal is a deep-tech startup founded by former Google Brain & X engineers
Ah. That explains it. "X" here also refers to Google, not Twitter/Musk devs.
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#9The most important bits about what they do to make this happen:
> In addition to the causal self-attention integral to transformers, we also allow each query token to attend to a fixed number of “external memories”. These memories are stored in a non-differentiable cache. The choice of which memories to attend to is made using cosine similarity within each decoder layer and attention head.
[...]
> We create our external memories (at each layer) by passing those external contexts through our model, just like inference. Then we save the internal representations the model generated, and attend to them later.
What an extremely clever approach!
If, in a chatbot setting, you update the external memory cache during inference, this means the thing immediately retains memory of the discussion.
Maybe this is an alternative (quicker? more exact?) to LoRA finetuning, to give a foundational model some specific personality and experiental history?
Re: Infinite Context LLMs: Going Beyond RAG with Extended Minds
#10With almost all of these papers, "RAG" is mentioned as a well-defined, absolute strategy. Is there an agreed upon implementation for it? Because as I have been building my own implementation, I have found that finding the right things to retrieve and augment the prompt with is incredibly challenging. It seems to me that a great RAG would almost always outperform other strategies because it will be like giving the stu…
the main problem with RAG IMHO is that. What if the note(s) they passed you have nothing to do with the answer you’re trying to respond to.
Getting RAG to work right depending on the context is tricky. I like RAG fusion approach with rerankers.