Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG at scale: Synchronizing and ingesting billions of text embeddings

medium.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#51

Post by juxtaposicion »

esafak wrote: post #22
juxtaposicion wrote: post #5

We’re also building billion-scale pipeline for indexing embeddings. Like the author, most of our pain has been scaling. If you only had to do millions, this whole pipeline would be a 100 LoC. but billions? Our system is at 20k LoC and growing. The biggest surprise to me here is using Weavite at the scale of billions — my understanding was that this would require tremendous memory requirements (of order a TB in RAM) w…

What kind of retrieval performance are you observing with Lance?

For a "small" dataset of 50M and 0.5TB in size with 20 results get around 50-100ms.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#52

Post by dluc »

CharlieDigital wrote: post #33
dluc wrote: post #9

We are also developing an open-source solution for those who would like to test it out and/or contribute, it can be consumed as a web service, or embedded into .NET apps. The project is codenamed "Semantic Memory" (available in GitHub) and offers customizable external dependencies, such as using Azure Queues, RabbitMQ, or other alternatives, and options for Azure Cognitive Search, Qdrant (with plans to include Weavia…

Why .NET apps specifically?

Multiple reasons, some are subjective as usual in these choices. Customers, performance, existing SK community, experience, etc.

However, the recommended use is running it as a web service, so from a consumer perspective the language doesn't really matter.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#53

Post by dluc »

bradneuberg wrote: post #38
dluc wrote: post #9

We are also developing an open-source solution for those who would like to test it out and/or contribute, it can be consumed as a web service, or embedded into .NET apps. The project is codenamed "Semantic Memory" (available in GitHub) and offers customizable external dependencies, such as using Azure Queues, RabbitMQ, or other alternatives, and options for Azure Cognitive Search, Qdrant (with plans to include Weavia…

Really interesting library. Is anyone aware of something similar but hooked into Google Cloud infra instead of Azure?

we could easily add that if there's interest, e.g. using Pub/Sub and Cloud Storage. If there are .NET libraries, should be straightforward implementing some interfaces. Similar considerations for the inference part, embedding and text generation.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#54

Post by avthar »

wanderingmind wrote: post #18

Are there any good implementations of using RAG within postgresql ecosystem? I have seen blogposts from supabase[0] and timescale db[1] but not a full fledged project. The full text search is very good within postgres at the moment and having semantic search within the same ecosystem is quiet helpful atleast for simple usecases. [0] https://supabase.com/docs/guides/database/extensions/pgvecto... [1] https://www.times…

Timescale recently released Timescale Vector [0] a scalable search index (DiskANN) and efficient time-based vector search, in addition to all capabilities of pgvector and vanilla PostgreSQL. We plan to add the document processing and embedding creation capabilities you discuss into our Python client library [1] next, but Timescale Vector integrates with LangChain and LlamaIndex today [2], which both have document chunking and embedding creation capabilities. (I work on Timescale Vector)

[0]: https://www.timescale.com/blog/how-we-made-postgresql-the-be... [1]: https://github.com/timescale/python-vector [2]: https://www.timescale.com/ai/#resources

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#55

Post by wanderingmind »

ddematheu wrote: post #29
wanderingmind wrote: post #21

Earlier quoted context omitted.

what i'm looking for is a neat python library (or equivalent) that integrates end to end say with postgres/pgvector using sqlalchemy, enables parallel processing of large number of documents, create interfaces for embeddings using openai/ollama etc. It looks like FastRAG [0] from intel looks close to what i'm envisioning but it doesnt appear to have integration to postgres ecosystem yet i guess. [0] https://github.co…

Through the platform (Neum AI) we support the ability to do this with Postgres, it is just a cloud platform so not a python library. Curious on what type of customization are you looking to add that you would want something like a library?

We need something we can orchestrate and control locally and be able make changes if need be. The GUI based interface is good for more mature workflows but our workflows are constantly evolving and requires tweaking that its hard to do with GUI and web interface

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#56

Post by vimota »

ddematheu wrote: post #42
vimota wrote: post #35

Thanks for writing this up! I'm working on a very similar service ( https://embeddingsync.com/ ) and I implemented almost the same as you've described here, but using a poll-based stateful workflow model instead of queueing. The biggest challenge - which I haven't solved as seamlessly as I'd like - is supporting updates / deletes in the source. You don't seem to discuss it in this post, does Neum handle that?

Co-author of the article here. We do support updates for some sources. Deletes not yet. For some sources we do polling which is then dumped on the queues. For other we have listeners that subscribe to changes. What are the challenges you are facing in supporting this?

Similar to you, for polling you only see new data not the deletion events so I can't delete embeddings unless I keep track of state and do a diff. To properly support that you/I would need effectively CDC, which gets more complex for arbitrary / self-serve databases.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#57

Post by derekperkins »

dluc wrote: post #53
bradneuberg wrote: post #38

Earlier quoted context omitted.

Really interesting library. Is anyone aware of something similar but hooked into Google Cloud infra instead of Azure?

we could easily add that if there's interest, e.g. using Pub/Sub and Cloud Storage. If there are .NET libraries, should be straightforward implementing some interfaces. Similar considerations for the inference part, embedding and text generation.

GCP also has a hosted vector db https://cloud.google.com/vertex-ai/docs/vector-search/overvi...
Post reply on HN