Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PageIndex – Vectorless RAG

github.com

Post reply on HN
Print view View original

21–30 of 147 posts

Re: Show HN: PageIndex – Vectorless RAG

#21

Post by agentcoops »

mosselman wrote: post #4

So if I understand this correctly it goes over every possible document with an LLM each time someone performs a search? I might have misunderstood of course. If so, then the use cases for this would be fairly limited since you'd have to deal with lots of latency and costs. In some cases (legal documents, medical records, etc) it might be worth it though. An interesting alternative I've been meaning to try out is inve…

I’ve been working on RAG systems a lot this year and I think one thing people miss is that often for internal RAG efficiency/latency is not the main concern. You want predictable, linear pricing of course, but sometimes you want to simply be able to get a predictably better response by throwing a bit more money/compute time at it.

It’s really hard to get to such a place with standard vector-based systems, even GraphRag. Because it relies on summaries of topic clusters that are pre-computed, if one of those summaries is inaccurate or none of the summaries deal with your exact question, that will never change during query processing. Moreover, GraphRag preprocessing is insanely expensive and precisely does not scale linearly with your dataset.

TLDR all the trade-offs in RAG system design are still being explored, but in practice I’ve found the main desired property to be “predictably better answer with predictably scaling cost” and I can see how similar concerns got OP to this design.

Re: Show HN: PageIndex – Vectorless RAG

#22

Post by thatjoeoverthr »

There's good reasons to do this. Embedding similarity is _not_ a reliable method of determining relevance.

I did some measurements and found you can't even really tell if two documents are "similar" or not. Here: https://joecooper.me/blog/redundancy/

One common way is to mix approaches. e.g. take a large top-K from ANN on embeddings as a preliminary shortlist, then run a tuned LLM or cross encoder to evaluate relevance.

I'll link here these guys' paper which you might find fun: https://arxiv.org/pdf/2310.08319

At the end of the day you just want a way to shortlist and focus information that's cheaper, computationally, and more reliable, than dumping your entire corpus into a very large context window.

So what we're doing is fitting the technique to the situation. Price of RAM; GPU price; size of dataset; etc. The "ideal" setup will evolve as the cost structure and model quality evolves, and will always depend on your activity.

But for sure, ANN-on-embedding as your RAG pipeline is a very blunt instrument and if you can afford to do better you can usually think of a way.

Re: Show HN: PageIndex – Vectorless RAG

#23

Post by [deleted] »

mosselman wrote: post #4

So if I understand this correctly it goes over every possible document with an LLM each time someone performs a search? I might have misunderstood of course. If so, then the use cases for this would be fairly limited since you'd have to deal with lots of latency and costs. In some cases (legal documents, medical records, etc) it might be worth it though. An interesting alternative I've been meaning to try out is inve…

[deleted]

Re: Show HN: PageIndex – Vectorless RAG

#24

Post by falcor84 »

mikeve wrote: post #17

Not sure if I fully understand it, but this seems highly inefficient? Instead of using embeddings which are easy to make a cheap to compare, you use summarized sections of documents and process them with an LLM? LLM's are slower and more expensive to run.

If this is used as an important tool call for an AI agent that preforms many other calls, then it's likely that the added cost and latency would be negligible compared to the benefit of significantly improved retrieval. As an analogy, for a small task you're often ok with just going over the first few search results, but to prepare for a large project, you might want to spend an afternoon researching.

Re: Show HN: PageIndex – Vectorless RAG

#25

Post by physicsguy »

agentcoops wrote: post #21
mosselman wrote: post #4

So if I understand this correctly it goes over every possible document with an LLM each time someone performs a search? I might have misunderstood of course. If so, then the use cases for this would be fairly limited since you'd have to deal with lots of latency and costs. In some cases (legal documents, medical records, etc) it might be worth it though. An interesting alternative I've been meaning to try out is inve…

I’ve been working on RAG systems a lot this year and I think one thing people miss is that often for internal RAG efficiency/latency is not the main concern. You want predictable, linear pricing of course, but sometimes you want to simply be able to get a predictably better response by throwing a bit more money/compute time at it. It’s really hard to get to such a place with standard vector-based systems, even GraphR…

Yes, in the use case we're doing it's been diagnosis of issues, and draws on documents in that. the latency doesn't matter because it's all done before the diagnosis is raised to the customer.

Re: Show HN: PageIndex – Vectorless RAG

#28

Post by jdthedisciple »

mosselman wrote: post #4

So if I understand this correctly it goes over every possible document with an LLM each time someone performs a search? I might have misunderstood of course. If so, then the use cases for this would be fairly limited since you'd have to deal with lots of latency and costs. In some cases (legal documents, medical records, etc) it might be worth it though. An interesting alternative I've been meaning to try out is inve…

> let an LLM note all of the possible questions that you can answer

What does this even mean? At what point do you know you have all of them?

Humans are quite ingenious coming up with new, unique questions in my observation, whereas LLMs have a hard time replicating those efficiently.

Re: Show HN: PageIndex – Vectorless RAG

#29

Post by summarity »

Qwuke wrote: post #15
mosselman wrote: post #4

So if I understand this correctly it goes over every possible document with an LLM each time someone performs a search? I might have misunderstood of course. If so, then the use cases for this would be fairly limited since you'd have to deal with lots of latency and costs. In some cases (legal documents, medical records, etc) it might be worth it though. An interesting alternative I've been meaning to try out is inve…

>An interesting alternative I've been meaning to try out is inverting this flow. Instead of using an LLM at time of searching to find relevant pieces to the query, you flip it around: at time of ingesting you let an LLM note all of the possible questions that you can answer with a given text and store those in an index. You may already know of this one, but consider giving Google LangExtract a look. A lot of companie…

This is just a variation of index time HyDE (Hypothetical Document Embedding). I used a similar strategy when building the index and search engine for findsight.ai
Post reply on HN