Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evaluating modular RAG with reasoning models

kapa.ai

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Evaluating modular RAG with reasoning models

#11

Post by laichzeit0 »

bauefi wrote: post #5
mkesper wrote: post #3

Latency must be brutal here. This will not be possible for any chat application, I guess.

It depends on how you do retrieval. If you just use dense embeddings for example you can get the latency of one search query down to maybe something like 400ms. In that case multiple sequential look ups would be ok but your embeddings need to be good enough of course

It's not just the retrieval, tool calls entail another call to the LLM (ToolMessage) and possibly the result will then require other tool calls. Massive latency.

Re: Evaluating modular RAG with reasoning models

#12

Post by serjester »

We tried something similar and found much better results with o1 pro than o3 mini. RAG seems to require a level of world knowledge that the mini models don’t have.

This comes at the cost of significantly higher latency and cost. But for us, answer quality is a much higher priority.

Re: Evaluating modular RAG with reasoning models

#13

Post by EngineeringStuf »

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query?

If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk.

That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked.

The trick, of course, is chunking it correctly and generating the right questions. But in both cases I would look to the LLM to do that.

Happy to recommend some tips on semantically splitting documents using the LLM with really low token usage if you're interested.

Re: Evaluating modular RAG with reasoning models

#14

Post by emil_sorensen »

serjester wrote: post #12

We tried something similar and found much better results with o1 pro than o3 mini. RAG seems to require a level of world knowledge that the mini models don’t have. This comes at the cost of significantly higher latency and cost. But for us, answer quality is a much higher priority.

Super cool! Yep, a lot seems to get lost through distillation.

Re: Evaluating modular RAG with reasoning models

#16

Post by ekianjo »

EngineeringStuf wrote: post #13

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query? If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk. That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked. The trick, of course, is chunking it correctly and gen…

> time and generate possible questions from the LLM based on the context of the current semantically split chunk.

Possible but very compute intensive. Imagine if you have hundreds of thousands of chunks...

Re: Evaluating modular RAG with reasoning models

#17

Post by TechDebtDevin »

EngineeringStuf wrote: post #13

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query? If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk. That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked. The trick, of course, is chunking it correctly and gen…

> Happy to recommend some tips on semantically splitting documents using the LLM with really low token usage if you're interested.

Go on please :)

Re: Evaluating modular RAG with reasoning models

#19

Post by SubiculumCode »

I found it interesting the parts that discussed current limitations of llm's understanding of tools, despite apparent reasoning abilities, it didn't seem to have an intuitive understanding of when to use the specific search tools.

I wonder whether this would benefit from a fine tuned llm module for that specific step, or even by providing a set of examples in the prompt of when to use what tool?

Re: Evaluating modular RAG with reasoning models

#20

Post by eternityforest »

serjester wrote: post #12

We tried something similar and found much better results with o1 pro than o3 mini. RAG seems to require a level of world knowledge that the mini models don’t have. This comes at the cost of significantly higher latency and cost. But for us, answer quality is a much higher priority.

RAG seems to work with 0.5 and 1.5B models just fine a lot of the time, it just can't handle anything that's not directly spelled out in the documents.

Or, at least it seems to in the limited amount of testing I did in a weekend. I'm an embedded dev without any real AI experience or an actual use case for building a RAG at the moment.

Post reply on HN