Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

61–70 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#61

Post by alansaber »

jascha_eng wrote: post #3

I have a RAG setup that doesn't work on documents but other data points that we use for generation (the original data is call recordings but it is heavily processed to just a few text chunks). Instead of a reranker model we do vector search and then simply ask GPT-5 in an extra call which of the results is the most relevant to the input question. Is there an advantage to actual reranker models rather than using a gen…

I think you should do both in parallel, rather than sequentially. Main reason is vector scoring could cut off something that an LLM will score as relevant

Re: Production RAG: what I learned from processing 5M+ documents

#62

Post by remz14 »

roze_sha wrote: post #30
prettyblocks wrote: post #27

Earlier quoted context omitted.

Which do you suggest?

https://huggingface.co/spaces/mteb/leaderboard

You should use RTEB instead. See here for why: https://huggingface.co/blog/rteb

Here is that leaderboard https://huggingface.co/spaces/mteb/leaderboard?benchmark_nam...

Voyage-3-large seems like SOTA right now

Re: Production RAG: what I learned from processing 5M+ documents

#63

Post by pamelafox »

catmanjan wrote: post #58
pamelafox wrote: post #56

Earlier quoted context omitted.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

I'd love to work with Azure search but because copilot with external items has been made so cheap it's hard to justify...

Do you mean that you're using the Copilot indexer for Sharepoint docs? https://learn.microsoft.com/en-us/microsoftsearch/semantic-i...

AI Search team's been working with the Sharepoint team to offer more options, so that devs can get best of both worlds. Might have some stuff ready for Ignite (mid November).

Re: Production RAG: what I learned from processing 5M+ documents

#64

Post by pamelafox »

alansaber wrote: post #59
pamelafox wrote: post #56

Earlier quoted context omitted.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

That is concerning given that pure vector search is terrible outside of abstractions

I know :( But I think vector DBs and vector search got so hyped that people thought you could switch entirely over to them. Lots of APIs and frameworks also used "vector store" as the shorthand for "retrieval data source", which didn't help.

That's why I write blog posts like https://blog.pamelafox.org/2024/06/vector-search-is-not-enou...

Re: Production RAG: what I learned from processing 5M+ documents

#67

Post by pietz »

Trias11 wrote: post #38
pietz wrote: post #37

I find it interesting that so many services and tools were investigated except for embedding models. I would have thought that's one of the biggest levers.

they just grabbed the better one (3-large) right off the bat. 6x cost to 3-small, but it's still tiny.

But the model is like 18 months old. and recently we've seen big leaps on MTEB. Not sure how well those translate to reality, but I'm a little surpised this wasn't worth looking into.

Re: Production RAG: what I learned from processing 5M+ documents

#68

Post by osigurdson »

pamelafox wrote: post #56
siva7 wrote: post #39

Earlier quoted context omitted.

Boy, that should not be the concern of the end user (developer) but those implementing RAG solutions as a service at Amazon, Microsoft, Openai and so on.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

Are you using Elasticsearch behind the scenes?

Re: Production RAG: what I learned from processing 5M+ documents

#69

Post by osigurdson »

pamelafox wrote: post #64
alansaber wrote: post #59

Earlier quoted context omitted.

That is concerning given that pure vector search is terrible outside of abstractions

I know :( But I think vector DBs and vector search got so hyped that people thought you could switch entirely over to them. Lots of APIs and frameworks also used "vector store" as the shorthand for "retrieval data source", which didn't help. That's why I write blog posts like https://blog.pamelafox.org/2024/06/vector-search-is-not-enou...

It is almost like embeddings are a technology from the olden days.

Re: Production RAG: what I learned from processing 5M+ documents

#70

Post by latchkey »

arcanemachiner wrote: post #35
hatmanstack wrote: post #31

Not here to schlep for AWS but S3 Vectors is hands down the SOTA here. That combined with a Bedrock Knowledge Base to handle Discovery/Rebalance tasks makes for the simplest implementation on the Market. Once Bedrock KB backed by S3 Vectors is released from Beta it'll eat everybody's lunch.

Shill, not schlep. I'm correcting you less out of pedantry, and more because I find the correct term to be funny.

Especially now that if you google the word schlep, the first result is now something totally different than what you'd expect.
Post reply on HN