Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production RAG: what I learned from processing 5M+ documents

blog.abdellatif.io

Post reply on HN
Print view View original

111–116 of 116 posts

Re: Production RAG: what I learned from processing 5M+ documents

#111

Post by swyx »

tifa2up wrote: post #93
swyx wrote: post #91

> LLM: GPT 4.1 -> GPT 5 -> GPT 4.1, covered by Azure credits whats this roundtrip? also the chronology of the LLM (4.1) doesnt match the rest of the stack (text-embedding-large-3), feels weird

OP. We migrated to GPT-5 when it came out but found that it performs worse than 4.1 when you pass lots of context (up to 100K tokens in some cases). We found that it: a) has worse instruction following; doesn't follow the system prompt b) produces very long answers which resulted in a bad ux c) has 125K context window so extreme cases resulted in an error Again, these were only observed in RAG when you pass lots of c…

love the share, ty

Re: Production RAG: what I learned from processing 5M+ documents

#112

Post by sangwook »

hatmanstack wrote: post #31

Not here to schlep for AWS but S3 Vectors is hands down the SOTA here. That combined with a Bedrock Knowledge Base to handle Discovery/Rebalance tasks makes for the simplest implementation on the Market. Once Bedrock KB backed by S3 Vectors is released from Beta it'll eat everybody's lunch.

S3 Vectors is great in terms of cost. But it provides around 500ms median query latency for 1M vectors, unlike other vector stores. And it does not support keyword search and sparse vectors. So I think it is better to choose which vector store to use based on your requirements.

Re: Production RAG: what I learned from processing 5M+ documents

#113

Post by crassT »

manishsharan wrote: post #2

Thanks for sharing. TIL about rerankers. Chunking strategy is a big issue. I found acceptable results by shoving large texts to to gemini flash and have it summarize and extract chunks instead of whatever text splitter I tried. I use the method published by Anthropic https://www.anthropic.com/engineering/contextual-retrieval i.e. include full summary along with chunks for each embedding. I also created a tool to enab…

I made a startup, https://tokencrush.ai/, to do just this.

I've struggled to find a target market though. Would you mind sharing what your use case is? It would really help give me some direction.

Re: Production RAG: what I learned from processing 5M+ documents

#115

Post by catmanjan »

pamelafox wrote: post #63
catmanjan wrote: post #58

Earlier quoted context omitted.

I'd love to work with Azure search but because copilot with external items has been made so cheap it's hard to justify...

Do you mean that you're using the Copilot indexer for Sharepoint docs? https://learn.microsoft.com/en-us/microsoftsearch/semantic-i... AI Search team's been working with the Sharepoint team to offer more options, so that devs can get best of both worlds. Might have some stuff ready for Ignite (mid November).

No we have a Microsoft graph connector which inserts externalitems into graph, copilot is able to surface these, probably via the same semantic search database

The capability was there for years, but it was expensive - something like $0.60 per 1000 items indexed, then sometimes after copilot was added it became free for up to 50 million items, and now it's free for unlimited items - you just can't beat that for price... https://techcommunity.microsoft.com/blog/microsoft365copilot...

Re: Production RAG: what I learned from processing 5M+ documents

#116

Post by pietz »

jokethrowaway wrote: post #106
pietz wrote: post #98

My biggest RAG learning is to use agentic RAG. (Sorry for buzzword dropping) - Classic RAG: `User -> Search -> LLM -> User` - Agentic RAG: `User LLM Search` Essentially instead of having a fixed loop, you provide the search as a tool to the LLM, which does three things: - The LLM can search multiple times - The LLM can adjust the search query - The LLM can use multiple tools The combination of these three things has…

yes but the assistant often doesn't search when it should and very rarely does multiple search rounds (both on gpt5 or on claude sonnet 4.5, weaker models are even worse at tool calling)

Cannot confirm this. Both sound like prompting issues.

- Depends on your use case to let the model understand when and when not to use tools - gpt-5 s VERY persistent and often searches more than 10 times in a single run depending on the results.

We're using pydantic AI where the entire Agent loop is taken care of by the framework. Highly recommend.

Post reply on HN