Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

I’ve spent the past year and a half constructing a Retrieval Augmented Generation (RAG) chatbot for a Fortune 500 manufacturing company, integrating over 50 million records across a dozen databases. Despite that scale, the system can return relevant info in 10–30 seconds, and it’s now at 90% five-star user approval internally.

After tons of trial and error—embedding huge datasets, mixing vector + text search, handling concurrency, and dodging hallucinations, I decided to document it all in a book. It’ll be live on Manning.com’s Early Access soon (March 27th). If you’re tackling large-scale RAG or have questions about my approach (the struggles, the successes), feel free to ask. I’m happy to share lessons, config ideas, or gotchas so you can avoid the pitfalls I hit along the way.

do share the db you used for starters and the overall stack like MERN or MEAN or Firestore BAAS or Supabase or something extremely different..

Focusing on the RA part of the RAG, which techniques or tools would you say contributed the most to the quality of the results? What sort of tradeoffs did you have to make?

How did you measure success of the RAG solution beyond five-star user approvals? Are there any critical metrics that determine success or failure?

How to start small in a very specific area: 1. Write down a list of 10 to 20 questions that you want your RAG to be able to answer. Then write down the correct answers next to each question. If you don't know the correct answers, ask a subject matter expert. Build a RAG chatbot that can answer those questions first. 2. You have to start out using tools that are capable of growing with you. For example, don't use ChromaDB because it can only fit in memory. Use services that can scale up. 3. I cover this in my book, you can use the 10 to 20 questions as integration tests and run them every time you commit code to make sure everything still works. 4. How to feed with data continuously... do you mean keeping your databases updated? You can write a script to check for regulations published in the last week, and only upload those regulations to your database.

I hope all that helps, let me know if you have any other questions!