Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
Memory-constrained environments.
Adaptive RAG – dynamic retrieval methods adjustment
31–40 of 40 posts
Re: Adaptive RAG – dynamic retrieval methods adjustment
#32Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
yes - private data, real-time data, curated data, citations with no hallucinations, RAG on tabular data, RAG on video, RAG on hierarchical mixed data, RAG over a graph
Re: Adaptive RAG – dynamic retrieval methods adjustment
#33This seems similar to building a RAG router (1) to perform dynamic retrieval/querying over data.
After getting hundreds of questions on my Interactive Resume AI chatbot (2), I've found the user queries can be categorized as: greeting, professional skills question, professional experience question, personal/hobby question and common interview question.
I am currently working on building a RAG router to help improve the quality of Q&A responses. I currently use gpt3.5 turbo without any special RAG techniques and the quality is lacking on performing Q&A over my resume and Q&A csv file. GPT4 works well but is too expensive.
1. https://docs.llamaindex.ai/en/stable/examples/low_level/rout... 2. https://jon-olson.com/resume_ai
Re: Adaptive RAG – dynamic retrieval methods adjustment
#34Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.
Well I guess the assumption is that you can just fit everything into the context and there'd be nothing to retrieve anymore.
Re: Adaptive RAG – dynamic retrieval methods adjustment
#35Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
Memory-constrained environments.
If you're running a 1M context LLM I assume you are well past memory constraints since the weights alone are 10s of GBs.
Re: Adaptive RAG – dynamic retrieval methods adjustment
#36Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.
But this is just current state. Token costs continue to go down and contexts will continue to get larger.
Re: Adaptive RAG – dynamic retrieval methods adjustment
#37Earlier quoted context omitted.
Could you elaborate a bit more about how that would work in practice?
Sure, if you're running a customer service chatbot, you can ask customers what the problem is, then start running rag async to populate a proper context for a smart LLM, and have the chatbot continue asking some questions to clarify details to give the background RAG process time to fetch data and run a quick summary, then have the chatbot give some indication it's thinking, run the full context query on the smart LL…
I see what you're saying, but you're assuming that consumer products are always chatbots (and that a small language model can buy time interacting with the user while possibly providing additional context). That being said, I would be interested to see such a system in practice - any examples you can point me to? My more general point was not chat-related; much of the research around RAG seems to use LLMs to parse or route the user's query, improve retrieval, etc. which doesn't often work in practice.
Re: Adaptive RAG – dynamic retrieval methods adjustment
#38Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?
I'm still not sold on recall at such large context window sizes. It's easy for an LLM to find a needle in a haystack, but in most RAG use-cases it's like finding a needle in a stack of needles, and the benchmarks don't really reflect that. There's also the speed and cost implications of dumping millions of tokens into a prompt - it's prohibitively slow and expensive right now.
Re: Adaptive RAG – dynamic retrieval methods adjustment
#39Earlier quoted context omitted.
Sure, if you're running a customer service chatbot, you can ask customers what the problem is, then start running rag async to populate a proper context for a smart LLM, and have the chatbot continue asking some questions to clarify details to give the background RAG process time to fetch data and run a quick summary, then have the chatbot give some indication it's thinking, run the full context query on the smart LL…
I see what you're saying, but you're assuming that consumer products are always chatbots (and that a small language model can buy time interacting with the user while possibly providing additional context). That being said, I would be interested to see such a system in practice - any examples you can point me to? My more general point was not chat-related; much of the research around RAG seems to use LLMs to parse or…
This is where the opportunity for creativity comes in. You could allow a chat based refinement to search queries, or provide popup refinement buttons that narrow the search space, and build the search results iteratively rather than the old paradigm of "search" -> "results"