Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG at scale: Synchronizing and ingesting billions of text embeddings

medium.com

Post reply on HN
Print view View original

21–30 of 57 posts

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#21

Post by wanderingmind »

losteric wrote: post #19
wanderingmind wrote: post #18

Are there any good implementations of using RAG within postgresql ecosystem? I have seen blogposts from supabase[0] and timescale db[1] but not a full fledged project. The full text search is very good within postgres at the moment and having semantic search within the same ecosystem is quiet helpful atleast for simple usecases. [0] https://supabase.com/docs/guides/database/extensions/pgvecto... [1] https://www.times…

Isn't RAG "just" dynamically injecting relevant text in a prompt? What more would one implement to achieve RAG, beyond using Postgres' built in full text or knn search?

what i'm looking for is a neat python library (or equivalent) that integrates end to end say with postgres/pgvector using sqlalchemy, enables parallel processing of large number of documents, create interfaces for embeddings using openai/ollama etc. It looks like FastRAG [0] from intel looks close to what i'm envisioning but it doesnt appear to have integration to postgres ecosystem yet i guess.

[0] https://github.com/IntelLabs/fastRAG

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#22

Post by esafak »

juxtaposicion wrote: post #5

We’re also building billion-scale pipeline for indexing embeddings. Like the author, most of our pain has been scaling. If you only had to do millions, this whole pipeline would be a 100 LoC. but billions? Our system is at 20k LoC and growing. The biggest surprise to me here is using Weavite at the scale of billions — my understanding was that this would require tremendous memory requirements (of order a TB in RAM) w…

What kind of retrieval performance are you observing with Lance?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#23

Post by janalsncm »

ddematheu wrote: post #4
joewferrara wrote: post #2

This is a great article about the technical difficulties of building a RAG system at scale from an engineering perspective. Performance is about speed and compute. A topic that is not addressed is how to evaluate a RAG system where performance is about whether the RAG system is retrieving the correct context and answering questions accurately. A RAG system should be built so that the different parts (retriever, embed…

Co-author of the article here. You are right. Retrieval accuracy is important as well. From an accuracy perspective, any tools you have found useful in helping validate retrieval accuracy? In our current architecture, all the different pieces within the RAG ingestion pipeline are modifiable to be able to improve loading, chunking and embedding. As part of our development process, we have started to enable other tools…

> From an accuracy perspective, any tools you have found useful in helping validate retrieval accuracy?

You’ll probably want to start with the standard rank-based metrics like MRR, nDCG, and precision/recall@K.

Plus if you’re going to spend $$$ embedding tons of docs you’ll want to compare to a “dumb” baseline like bm25.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#24

Post by typest »

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system.

Is everyone currently reinventing search from first principles?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#25

Post by omneity »

bryan0 wrote: post #10
juxtaposicion wrote: post #5

We’re also building billion-scale pipeline for indexing embeddings. Like the author, most of our pain has been scaling. If you only had to do millions, this whole pipeline would be a 100 LoC. but billions? Our system is at 20k LoC and growing. The biggest surprise to me here is using Weavite at the scale of billions — my understanding was that this would require tremendous memory requirements (of order a TB in RAM) w…

we've been using pgvector at the 100M scale without any major problems so far, but I guess it depends on your specific use case. we've also been using elastic search dense vector fields which also seems to scale well, but of course its pricey but we already have it in our infra so works well.

What size are your embeddings?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#26

Post by mrfox321 »

typest wrote: post #24

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?

Your intuition on search being implemented is correct.

It's still TBD on whether these new generations of language models will democratize search on bespoke corpuses.

There's going to be a lot of arbitrary alchemy and tribal knowledge...

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#27

Post by dkatz23238 »

What statistics/metrics are used to evaluate RAG systems? Is there any paper that systematically compares different RAG methods (chunkings, models, ect)? I would assume that such metric would be similar to something used for evaluating summarization or question and answering but I am curious to know if there are specific methods/metrics used to evaluate RAG systems.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#28

Post by ddematheu »

typest wrote: post #24

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?

To some degree. The amount of data that will be brought into search solutions will be enormous, seems like a good time to try to reimagine what that process might look like

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#29

Post by ddematheu »

wanderingmind wrote: post #21
losteric wrote: post #19

Earlier quoted context omitted.

Isn't RAG "just" dynamically injecting relevant text in a prompt? What more would one implement to achieve RAG, beyond using Postgres' built in full text or knn search?

what i'm looking for is a neat python library (or equivalent) that integrates end to end say with postgres/pgvector using sqlalchemy, enables parallel processing of large number of documents, create interfaces for embeddings using openai/ollama etc. It looks like FastRAG [0] from intel looks close to what i'm envisioning but it doesnt appear to have integration to postgres ecosystem yet i guess. [0] https://github.co…

Through the platform (Neum AI) we support the ability to do this with Postgres, it is just a cloud platform so not a python library.

Curious on what type of customization are you looking to add that you would want something like a library?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#30

Post by dluc »

ddematheu wrote: post #16
dluc wrote: post #14

Earlier quoted context omitted.

Currently we use LLMs to generate a summary, used as an additional chunk. As you might guess, this can take time, so we postpone the summarization at the end (the current default pipeline is: extract, partition, gen embedding, save embeddings, summarize, gen embeddings (of the summary), save emb) Initial tests though are showing that summaries are affecting the quality of answers, so we'll probably remove it from the…

Makes sense. Interesting on the fact that summaries affect quality sometimes. For synthetic data scenarios are you also thinking about synthetic queries over the data? (Try to predict which chunks might be more used than others)

yes, queries and also planning.

For instance, given the user "ask" (which could be any generic message in a copilot), decide how to query one or multiple storages. Ultimately, companies and users have different storages, and a few can be indexed with vectors (and additional fine tuned models). But there's a lot of "legacy" structured data accessible only with SQL and similar languages, so a "planner" (in the SK sense of planners) could be useful to query vector indexes, text indexes and knowledge graphs, combining the result.

Post reply on HN