Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adaptive RAG – dynamic retrieval methods adjustment

arxiv.org

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Adaptive RAG – dynamic retrieval methods adjustment

#21

Post by CuriouslyC »

whakim wrote: post #20

From a consumer perspective, this is a super interesting paper because it touches on one of the fundamental issues with most RAG beyond the toy case - that you need to do different stuff depending on what the user is asking for. You also (usually) can't just ask because most users don't know that LLMs are bad at math or semantic search won't be sufficient to answer questions that involve enumeration or totality. And…

That depends on the model, you can run stuff in parallel and sometimes keep everything timely. You shouldn't be waiting till the last second to start running rag, you can be pre-emptively building context based on the current chat (like a human does) so that you've already got stuff summarized and ready to fire off when the final prompt does come.

Think about how a human will draw out a conversation around answering a question and use delaying words and phrases to let them continue answering when they don't have the solution fully formulated. LLMs can use the same tactic.

Re: Adaptive RAG – dynamic retrieval methods adjustment

#22

Post by CuriouslyC »

jillesvangurp wrote: post #19
machinelearning wrote: post #9

This is a simple version of the tree search approach that people suspect Q* is

Teaching LLMs how to search is probably going to be key to make them hallucinate far less. Most RAG approaches currently use simple vector searches to pull out information. Chat GPT actually is able to run Bing searches. And presumably Gemini uses Google's search. It's fairly clunky and unsophisticated currently. These searches are still relatively dumb. With LLMs not being half bad at remembering a lot of things, pr…

One big problem is that we've build search for humans, more specifically to advertise to them.

AI doesn't need a human search, it needs a "fact database" that can pull short factoids with a truth value, which could be a distribution based on human input. So for example, you might have the factoid "Donald Trump incited insurrection on January 6th" with a score of 0.8 (out of 1) with a 0.3 variance since people either tend to absolutely believe it or disbelieve it, with more people on the believing side.

Beyond that AI needs a "logical tools" database with short examples of their use that it can pull from for any given problem.

Re: Adaptive RAG – dynamic retrieval methods adjustment

#23

Post by seanc »

milliondreams wrote: post #17
boodleboodle wrote: post #4

Are we advertising papers on hackernews now?

I do find myself reading papers often for my work, and I share the once I find interesting or feel might have impact in future of my chosen domain. This is no advertisement, I don't know the authors or anyone related to the paper.

My father was a PhD psychologist and family therapist. He was on the witness stand during a custody case explaining a theory of personality when the cross-examining lawyer said scornfully "I'll bet you got that out of some book." To which my dad replied: "Why yes, in fact. In my profession, in order to learn things, we often read books."

Re: Adaptive RAG – dynamic retrieval methods adjustment

#24

Post by whakim »

CuriouslyC wrote: post #21
whakim wrote: post #20

From a consumer perspective, this is a super interesting paper because it touches on one of the fundamental issues with most RAG beyond the toy case - that you need to do different stuff depending on what the user is asking for. You also (usually) can't just ask because most users don't know that LLMs are bad at math or semantic search won't be sufficient to answer questions that involve enumeration or totality. And…

That depends on the model, you can run stuff in parallel and sometimes keep everything timely. You shouldn't be waiting till the last second to start running rag, you can be pre-emptively building context based on the current chat (like a human does) so that you've already got stuff summarized and ready to fire off when the final prompt does come. Think about how a human will draw out a conversation around answering…

Could you elaborate a bit more about how that would work in practice?

Re: Adaptive RAG – dynamic retrieval methods adjustment

#26

Post by CuriouslyC »

whakim wrote: post #24
CuriouslyC wrote: post #21

Earlier quoted context omitted.

That depends on the model, you can run stuff in parallel and sometimes keep everything timely. You shouldn't be waiting till the last second to start running rag, you can be pre-emptively building context based on the current chat (like a human does) so that you've already got stuff summarized and ready to fire off when the final prompt does come. Think about how a human will draw out a conversation around answering…

Could you elaborate a bit more about how that would work in practice?

Sure, if you're running a customer service chatbot, you can ask customers what the problem is, then start running rag async to populate a proper context for a smart LLM, and have the chatbot continue asking some questions to clarify details to give the background RAG process time to fetch data and run a quick summary, then have the chatbot give some indication it's thinking, run the full context query on the smart LLM, generate a summary answer then feed it back to the chat LLM and say "I may have found a solution to your problem" then switch to the response from the smart LLM.

Re: Adaptive RAG – dynamic retrieval methods adjustment

#27

Post by Narciss »

humansareok1 wrote: post #25

Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?

It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.

Re: Adaptive RAG – dynamic retrieval methods adjustment

#29

Post by eropple »

humansareok1 wrote: post #25

Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?

I think so. Mostly:

- Hosted solutions charge you for tokens. More tokens, more money. Keeping money in your pocket: generally recognized as rad.

- 1M tokens wouldn't hold the entire codebase I have open in my other window.

Re: Adaptive RAG – dynamic retrieval methods adjustment

#30

Post by chasd00 »

humansareok1 wrote: post #25

Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?

correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.
Post reply on HN