Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evaluating modular RAG with reasoning models

kapa.ai

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Evaluating modular RAG with reasoning models

#21

Post by emil_sorensen »

aantix wrote: post #18

When aggregating data from multiple systems, how do you handle the case of only searching against data chunks that the user is authorized to view? And if those permissions change?

We focus mainly on external use cases (e.g., helping companies like Docker and Monday.com deploy customer facing "Ask AI" assistants) so we don't run into much of that given all data is public.

For internal use cases that require user level permissions that's a freaking rabbit role. I recently heard someone describe Glean as a "permissions company" more so than a search company for that reason. :)

Re: Evaluating modular RAG with reasoning models

#22

Post by Foobar8568 »

eternityforest wrote: post #20
serjester wrote: post #12

We tried something similar and found much better results with o1 pro than o3 mini. RAG seems to require a level of world knowledge that the mini models don’t have. This comes at the cost of significantly higher latency and cost. But for us, answer quality is a much higher priority.

RAG seems to work with 0.5 and 1.5B models just fine a lot of the time, it just can't handle anything that's not directly spelled out in the documents. Or, at least it seems to in the limited amount of testing I did in a weekend. I'm an embedded dev without any real AI experience or an actual use case for building a RAG at the moment.

RAG is basically a fancy name to augment a prompt with data.

Companies are being sold they can augment their LLM with their unstructured massive dataset but it's all wishful thinking.

Re: Evaluating modular RAG with reasoning models

#23

Post by EngineeringStuf »

ekianjo wrote: post #16
EngineeringStuf wrote: post #13

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query? If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk. That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked. The trick, of course, is chunking it correctly and gen…

> time and generate possible questions from the LLM based on the context of the current semantically split chunk. Possible but very compute intensive. Imagine if you have hundreds of thousands of chunks...

The number of chunks would be the same regardless of either approach.

The generation of questions can be done out-of-band by a cheaper model.

Their current implementation approach seems to require some computation per request. It would be a balance to see which strategy provides the most value.

The speed of responses overall would be faster.

Re: Evaluating modular RAG with reasoning models

#24

Post by eternityforest »

bauefi wrote: post #5
mkesper wrote: post #3

Latency must be brutal here. This will not be possible for any chat application, I guess.

It depends on how you do retrieval. If you just use dense embeddings for example you can get the latency of one search query down to maybe something like 400ms. In that case multiple sequential look ups would be ok but your embeddings need to be good enough of course

These ultra fast embeddings are really cool, because you can just spam them at everything and it's pretty much instant.

I was able to get them to answer very simple questions without any vector database or pre indexing, just expanding the search query to synonyms, then using normal fulltext search, using embeddings to match article titles to the query, plus adding a few "Personality documents" that are always in every result set no matter what.

Then I do chunking on the fly based on similarity to to query.

Retrieval takes about 1 second on a CPU, but then the actual LLM call takes 10 to 40 seconds, because you need about 1500 bytes of context to consistently get something that has the answers in it... Not exactly useful at the moment on cheap consumer hardware but still very interesting.

https://huggingface.co/blog/static-embeddings

Re: Evaluating modular RAG with reasoning models

#25

Post by Workaccount2 »

Foobar8568 wrote: post #22
eternityforest wrote: post #20

Earlier quoted context omitted.

RAG seems to work with 0.5 and 1.5B models just fine a lot of the time, it just can't handle anything that's not directly spelled out in the documents. Or, at least it seems to in the limited amount of testing I did in a weekend. I'm an embedded dev without any real AI experience or an actual use case for building a RAG at the moment.

RAG is basically a fancy name to augment a prompt with data. Companies are being sold they can augment their LLM with their unstructured massive dataset but it's all wishful thinking.

Yeah, LLM capabilities are measured with fresh context windows, yet people want to use them with 50k, 100k, 500k tokens.

As you pack in more and more context the model's abilities really start to deteriorate.

The first 10k tokens are the juiciest, after that it just gets worse and worse.

Re: Evaluating modular RAG with reasoning models

#26

Post by eternityforest »

Workaccount2 wrote: post #25
Foobar8568 wrote: post #22

Earlier quoted context omitted.

RAG is basically a fancy name to augment a prompt with data. Companies are being sold they can augment their LLM with their unstructured massive dataset but it's all wishful thinking.

Yeah, LLM capabilities are measured with fresh context windows, yet people want to use them with 50k, 100k, 500k tokens. As you pack in more and more context the model's abilities really start to deteriorate. The first 10k tokens are the juiciest, after that it just gets worse and worse.

Oh wow, I was thinking 500 tokens was way too much, since I've only ever done anything programmatic with tiny models on CPUs....

Re: Evaluating modular RAG with reasoning models

#27

Post by Workaccount2 »

mvieira38 wrote: post #15
anonymousDan wrote: post #6

Is RAG any good for coding tasks?

For what it's worth, Anthropic decided against using RAG in Claude Code ( https://news.ycombinator.com/item?id=43164089 )

I used Claude 3.7 last night to write a program for building tooling code from legacy manufacturing files for cnc and electronics manufacturing. Basically it renders the old files visually (they are sorta like SVGs) and then a human can click through them to create the necessary measurements, which the program then indexes and stores for the user. It has a nice GUI with buttons, highlights your selections, graphically demarcates previous measurements, and shows a running list of calculated outputs based on your selections, which you can delete if incorrect. Then you click export it exports it in the correct modern Place File structure. Totally knocked my socks and feet off too.

There are no programs online which do this (lots of viewers, but not interpreters/converters), and I actually had gotten a quote for proprietary software that can do it, but is $1k/yr to use.

I _did not_ think claude would be able to do it, but thought I would give it a shot. It took 3 prompts to get to 95% of the way there. The last 5% was done by o3mini because Claude ran out of capacity for me.

Re: Evaluating modular RAG with reasoning models

#28

Post by triyambakam »

EngineeringStuf wrote: post #13

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query? If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk. That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked. The trick, of course, is chunking it correctly and gen…

So if the user submitted a question not already generated, would that be like a cache miss and it would instead fall back to a real time query?

Re: Evaluating modular RAG with reasoning models

#29

Post by EngineeringStuf »

triyambakam wrote: post #28
EngineeringStuf wrote: post #13

Am I correct in reading that the RAG pipeline runs in realtime in response to a user query? If so, then I would suggest that you run it ahead of time and generate possible questions from the LLM based on the context of the current semantically split chunk. That way you only need to compare the embeddings at query time and it will already be pre-sorted and ranked. The trick, of course, is chunking it correctly and gen…

So if the user submitted a question not already generated, would that be like a cache miss and it would instead fall back to a real time query?

Yes, but you could optimise the generated questions over time to reduce cache-misses.

Re: Evaluating modular RAG with reasoning models

#30

Post by serjester »

eternityforest wrote: post #20
serjester wrote: post #12

We tried something similar and found much better results with o1 pro than o3 mini. RAG seems to require a level of world knowledge that the mini models don’t have. This comes at the cost of significantly higher latency and cost. But for us, answer quality is a much higher priority.

RAG seems to work with 0.5 and 1.5B models just fine a lot of the time, it just can't handle anything that's not directly spelled out in the documents. Or, at least it seems to in the limited amount of testing I did in a weekend. I'm an embedded dev without any real AI experience or an actual use case for building a RAG at the moment.

That's essentially what an embedding model is - a smaller, faster model that's good at finding information quickly. Then you feed that to a larger, more powerful reasoning model to synthesize and you've invented RAG.
Post reply on HN