Evaluating modular RAG with reasoning models

Post by 3abiton » Thu, Feb 27, 2025, 1:07 AM UTC

emil_sorensen wrote: post #21

aantix wrote: post #18

When aggregating data from multiple systems, how do you handle the case of only searching against data chunks that the user is authorized to view? And if those permissions change?

We focus mainly on external use cases (e.g., helping companies like Docker and Monday.com deploy customer facing "Ask AI" assistants) so we don't run into much of that given all data is public. For internal use cases that require user level permissions that's a freaking rabbit role. I recently heard someone describe Glean as a "permissions company" more so than a search company for that reason. :)

> fine-tuning a model on tool usage could also allow it to gain familiarity with specific retrieval mechanisms.

I am curious if finetuning on specific usecases would outperform RAG approaches, assuming the data is static (say company documentation). I know there has been lots of posts on this, but yet to see quanitifications, especially with o3-mini.

Post by eternityforest » Thu, Feb 27, 2025, 12:14 PM UTC

serjester wrote: post #30

eternityforest wrote: post #20

Earlier quoted context omitted.

RAG seems to work with 0.5 and 1.5B models just fine a lot of the time, it just can't handle anything that's not directly spelled out in the documents. Or, at least it seems to in the limited amount of testing I did in a weekend. I'm an embedded dev without any real AI experience or an actual use case for building a RAG at the moment.

That's essentially what an embedding model is - a smaller, faster model that's good at finding information quickly. Then you feed that to a larger, more powerful reasoning model to synthesize and you've invented RAG.

In my limited weekend testing with just a CPU, the 1.5B model is the larger and more powerful model at the end!

I'm definitely excited to see what new applications are possible with NPUs, when we can run this stuff for real on stuff anyone other than enthusiasts can afford, without waiting 40 seconds.

Re: Evaluating modular RAG with reasoning models

Re: Evaluating modular RAG with reasoning models