Evaluating modular RAG with reasoning models
1–10 of 32 posts
Re: Evaluating modular RAG with reasoning models
#2Curious if anyone else has run similar experiments?
Re: Evaluating modular RAG with reasoning models
#3Latency must be brutal here. This will not be possible for any chat application, I guess.
Re: Evaluating modular RAG with reasoning models
#4Latency must be brutal here. This will not be possible for any chat application, I guess.
Yep even with a small bump in performance (which we only saw for a subset of coding questions), it wouldn't be worth the huge latency penalty. Though that will surely go down over time.
Re: Evaluating modular RAG with reasoning models
#5Latency must be brutal here. This will not be possible for any chat application, I guess.
It depends on how you do retrieval. If you just use dense embeddings for example you can get the latency of one search query down to maybe something like 400ms. In that case multiple sequential look ups would be ok but your embeddings need to be good enough of course
Re: Evaluating modular RAG with reasoning models
#6Is RAG any good for coding tasks?
Re: Evaluating modular RAG with reasoning models
#7Is RAG any good for coding tasks?
I'd say it's essential to provide whatever you're asking context. In fully local environments I've been able to integrate the responses directly without having the generalize -> generate -> de-generalize loop, highly increasing LLM's value for me.
Re: Evaluating modular RAG with reasoning models
#8Curious if anyone else has run similar experiments?
Yes. Our main finding was that o3 mini especially is great on paper but surprisingly hard to prompt, compared to non reasoning models.
I don't think it's a problem with reasoning, but rather with this specific model.
I also suspect that o3 mini is a rather small model and so it can lack useful knowledge for broad applications.
Especially for RAG, it seems that larger and fast models (e.g. gpt4o) perform better as of today.
Re: Evaluating modular RAG with reasoning models
#9Is RAG any good for coding tasks?
I'd say it's essential to provide whatever you're asking context. In fully local environments I've been able to integrate the responses directly without having the generalize -> generate -> de-generalize loop, highly increasing LLM's value for me.
Could you share more on your local setup please?
Re: Evaluating modular RAG with reasoning models
#10Curious if anyone else has run similar experiments?
Yes. Our main finding was that o3 mini especially is great on paper but surprisingly hard to prompt, compared to non reasoning models. I don't think it's a problem with reasoning, but rather with this specific model. I also suspect that o3 mini is a rather small model and so it can lack useful knowledge for broad applications. Especially for RAG, it seems that larger and fast models (e.g. gpt4o) perform better as of…
I suspect you're right here! Excited to get our hands on the non-distilled o3. :)