Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#71

Post by pamelafox »

osigurdson wrote: post #68
pamelafox wrote: post #56

Earlier quoted context omitted.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

Are you using Elasticsearch behind the scenes?

I believe that Azure AI Search currently uses lucene for BM25, hnswlib for vector search, and the Bing re-ranking model for semantic ranking. (So, no, it does not, though features are similar)

Re: Production RAG: what I learned from processing 5M+ documents

#72

Post by osigurdson »

yahoozoo wrote: post #41
tifa2up wrote: post #24

Earlier quoted context omitted.

OP. Reranking is a specialized LLM that takes the user query, and a list of candidate results, then re-sets the order based on which ones are more relevant to the query. Here's sample code: https://docs.cohere.com/reference/rerank

What is the difference between reranking versus generating text embeddings and comparing with cosine similarity?

Because LLMs are a lot smarter than embeddings and basic math. Think of the vector / lexical search as the first approximation.

Re: Production RAG: what I learned from processing 5M+ documents

#74

Post by icemanx »

goodev wrote: post #50
bityard wrote: post #45

I must be missing something, this says it can be self-hosted. But the first page of the self-hosting docs say you need accounts with no less than 6 (!) other third-party hosted services. We have very different ideas about the meaning of self-hosted.

I consider this to be good open source and I'm a happy user of their OSS offering. Want no hosted dependencies? Then go write it all in Rust.

that's a stupid take and shows lack of engineering experience

Re: Production RAG: what I learned from processing 5M+ documents

#75

Post by nl »

bityard wrote: post #45

I must be missing something, this says it can be self-hosted. But the first page of the self-hosting docs say you need accounts with no less than 6 (!) other third-party hosted services. We have very different ideas about the meaning of self-hosted.

You can self-host their code. I don't think there is any official definition of "self hosted" that this violates.

For example - if a "self hosted" service supports off-site backups is it self hosted or just well designed?

Re: Production RAG: what I learned from processing 5M+ documents

#76

Post by cipherself »

pamelafox wrote: post #56
siva7 wrote: post #39

Earlier quoted context omitted.

Boy, that should not be the concern of the end user (developer) but those implementing RAG solutions as a service at Amazon, Microsoft, Openai and so on.

At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-sear…

I am working on search but rather for text-to-image retrieval, nevertheless, I am curious if by that's all baked into Azure AI search you also meant synthetic query generation from the grandparent comment. If so, what's your latency for this? And do you extract structured data from the query? If so, do you use LLMs for that?

Moreover I am curious why you guys use bm25 over SPLADE?

Re: Production RAG: what I learned from processing 5M+ documents

#77

Post by cipherself »

hatmanstack wrote: post #31

Not here to schlep for AWS but S3 Vectors is hands down the SOTA here. That combined with a Bedrock Knowledge Base to handle Discovery/Rebalance tasks makes for the simplest implementation on the Market. Once Bedrock KB backed by S3 Vectors is released from Beta it'll eat everybody's lunch.

S3 Vectors is hands down the SOTA here

SOTA for what? Isn't it just a vector store?

Re: Production RAG: what I learned from processing 5M+ documents

#78

Post by mattfrommars »

Great read. But how do people land opportunities to work on exciting project as the author did? I've been trying to get into legal tech in LLM space but I've been unsuccessful.

Anyone here successfully transitioned into legal space? My gut always been legal to the space where LLM can really be useful, the first one is in programming.

Re: Production RAG: what I learned from processing 5M+ documents

#79

Post by torrmal »

we have been trying to make it so that people dont have to reinvent the wheel, over and over and over again, and have a very straight forward all batteries included that can scale to many millions of documents, combining the best of RAG with traditional search and parametric search, https://docs.mindsdb.com/mindsdb_sql/knowledge_bases/overvie... Would love your feedback.

Re: Production RAG: what I learned from processing 5M+ documents

#80

Post by Valk3_ »

derefr wrote: post #54
yahoozoo wrote: post #41

Earlier quoted context omitted.

What is the difference between reranking versus generating text embeddings and comparing with cosine similarity?

My understanding: If you generate embeddings (of the query, and of the candidate documents) and compare them for similarity, you're essentially asking whether the documents "look like the question." If you get an LLM to evaluate how well each candidate document follows from the query, you're asking whether the documents "look like an answer to the question." An ideal candidate chunk/document from a cosine-similarity…

I've been thinking about the problem of what to do if the answer to a question is very different to the question itself in embedding space. The KB method sounds interesting and not something I thought about, you sort work on the "document side" I guess. I've also heard of HYDE, the works on the query side, you generate hypothetical answers instead to the user query and look for documents that are similar to the answer, if I've understood it correctly.
Post reply on HN