Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG at scale: Synchronizing and ingesting billions of text embeddings

medium.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#31

Post by [deleted] »

losteric wrote: post #19
wanderingmind wrote: post #18

Are there any good implementations of using RAG within postgresql ecosystem? I have seen blogposts from supabase[0] and timescale db[1] but not a full fledged project. The full text search is very good within postgres at the moment and having semantic search within the same ecosystem is quiet helpful atleast for simple usecases. [0] https://supabase.com/docs/guides/database/extensions/pgvecto... [1] https://www.times…

Isn't RAG "just" dynamically injecting relevant text in a prompt? What more would one implement to achieve RAG, beyond using Postgres' built in full text or knn search?

[deleted]

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#32

Post by antupis »

ddematheu wrote: post #28
typest wrote: post #24

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?

To some degree. The amount of data that will be brought into search solutions will be enormous, seems like a good time to try to reimagine what that process might look like

Also this is search for LLM not for humans so optimal solution will be different. Or even with models it is not that hard to imagine that Mistral-8b will need different results than GPT4 which has 1.76 trillion parameters.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#33

Post by CharlieDigital »

dluc wrote: post #9

We are also developing an open-source solution for those who would like to test it out and/or contribute, it can be consumed as a web service, or embedded into .NET apps. The project is codenamed "Semantic Memory" (available in GitHub) and offers customizable external dependencies, such as using Azure Queues, RabbitMQ, or other alternatives, and options for Azure Cognitive Search, Qdrant (with plans to include Weavia…

Why .NET apps specifically?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#34

Post by antupis »

wanderingmind wrote: post #18

Are there any good implementations of using RAG within postgresql ecosystem? I have seen blogposts from supabase[0] and timescale db[1] but not a full fledged project. The full text search is very good within postgres at the moment and having semantic search within the same ecosystem is quiet helpful atleast for simple usecases. [0] https://supabase.com/docs/guides/database/extensions/pgvecto... [1] https://www.times…

Or generally what are good vector dbs have tried LlaMaindex, pinecone and milvus but all kinda sucked different way.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#35

Post by vimota »

Thanks for writing this up! I'm working on a very similar service (https://embeddingsync.com/) and I implemented almost the same as you've described here, but using a poll-based stateful workflow model instead of queueing.

The biggest challenge - which I haven't solved as seamlessly as I'd like - is supporting updates / deletes in the source. You don't seem to discuss it in this post, does Neum handle that?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#37

Post by isaacfung »

typest wrote: post #24

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?

Depends on what you mean by search. Do you consider all Question Answering as search?

Some questions require multi-hop reasoning or have to be decomposed into simpler subproblems. When you google a question, often the answer is not trivially included in the retrieved text and you have to process(filter irrelevant information, resolve conflicting information, extrapolate to cases not covered, align the same entities referred to with two different names, etc), forumate an answer for the original question and maybe even predict your intent based on your history to personalize the result or customize the result in the format you like(markdown, json, csv, etc).

Researchers have developed many different techniques to solve the related problems. But as LLMs are getting hyped, many people try to tell you LLM+vector store is all you need.

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#38

Post by bradneuberg »

dluc wrote: post #9

We are also developing an open-source solution for those who would like to test it out and/or contribute, it can be consumed as a web service, or embedded into .NET apps. The project is codenamed "Semantic Memory" (available in GitHub) and offers customizable external dependencies, such as using Azure Queues, RabbitMQ, or other alternatives, and options for Azure Cognitive Search, Qdrant (with plans to include Weavia…

Really interesting library.

Is anyone aware of something similar but hooked into Google Cloud infra instead of Azure?

Re: RAG at scale: Synchronizing and ingesting billions of text embeddings

#40

Post by zby »

typest wrote: post #24

It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?

I am convinced that we should teach the LLMs to use search as a tool instead of creating special search that is useful for LLMs. We now have a lot of search systems and LLMs can in theory use all kind of text interface, the only problem is with the limited context that LLMs can consume. But is is quite orthogonal to what kind of index we use for the search. In fact for humans it is also be useful that search returns limited chunks - we already have that with the 'snippets' that for example Google shows - we just need it to tweak a bit for them to be maybe two kind of snippets - shorter as they are now and longer.

You can use LLMs to do semantic search using a keyword search - by telling the LLM to come up with a good search term that would include all the synonymes. But if vector search in embeddings really gives better results than keyword search - then we should start using it in all the other search tools used by humans.

LLMs are the more general tool - so adjusting them to the more restricted search technology should be easier and quicker to do instead of doing it the other way around.

By the way - this prompted me to create my Opinionated RAG wiki: https://github.com/zby/answerbot/wiki

Post reply on HN