Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

news.ycombinator.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

#11

Post by tylersuard »

vlit20 wrote: post #7

How did you measure success of the RAG solution beyond five-star user approvals? Are there any critical metrics that determine success or failure?

We have tests in place to make sure every function, every search is working, otherwise things won't deploy. We do have a dashboard with usage etc., but I have not spent much time looking at it. I think we just count broad usage across the company as a success. And yes we consider one-stars and complaints to be a failure, but we get those less often, maybe once every 2 weeks, and it is almost always the user's fault. Does that answer your question?

Re: Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

#12

Post by tylersuard »

zergnick wrote: post #6

When you are dealing with documents with different structures, how to do the document chunking efficiently without losing important metadata?

First of all, great question.

Second, we use a search service, and vectors are treated as supplementary to the text search, so chunking doesn't matter as much. We will usually take an entire PDF page and embed that, no matter what structure the data on that page is. We do keep track of the name of the document and the page number. For SQL records, we just turn each record into a text string and embed that.

Re: Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

#13

Post by tylersuard »

karanveer wrote: post #3

do share the db you used for starters and the overall stack like MERN or MEAN or Firestore BAAS or Supabase or something extremely different..

Ok so the company has like 20 databases, plus over 100,000 pages of PDF catalogs. We tried using agents to query the company's SQL databases, but that took 30 seconds each call, and that is unacceptable because we wanted to return an answer to the user in 10 to 30 seconds. So what we ended up doing is, we created an Azure AI Search service and we made a different search index (like a collection) for each data source, one for each database and one for our repository of 100,000 pdf pages.

Our stack was just Python, Autogen for the agents, and as I mentioned Azure AI Search. We use Azure Web Apps for the backend, and OpenAI models for the generation. Great questions!

Re: Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

#14

Post by tylersuard »

decide1000 wrote: post #4

27th marked! What infrastructure did you use?

Great, thank you!

The main program is hosted on Azure Web Apps, the search is Azure AI Search, we use AutoGen for the agents, and we use OpenAI for the generation. Azure has a lot of tools that support AI and search, so we use those too.

Re: Ask HN: Lessons from Building a Fortune 500 RAG Chatbot (50M Records in 10–30s)

#16

Post by zergnick »

tylersuard wrote: post #12
zergnick wrote: post #6

When you are dealing with documents with different structures, how to do the document chunking efficiently without losing important metadata?

First of all, great question. Second, we use a search service, and vectors are treated as supplementary to the text search, so chunking doesn't matter as much. We will usually take an entire PDF page and embed that, no matter what structure the data on that page is. We do keep track of the name of the document and the page number. For SQL records, we just turn each record into a text string and embed that.

Thanks for your feedback! Could you share a bit about your team? I’m curious how many people are involved and what kinds of skills or roles are needed to make this happen.
Post reply on HN