41–50 of 57 posts
The openAI or replicate embeddings APIs are already a managed service... You would still need to self managing it all just into a different API.
And dealing with embeddings is the kind of fun work every engineer wants to do anyway.
Still a good article but very perplexing how the company can exist
Thanks for writing this up! I'm working on a very similar service ( https://embeddingsync.com/ ) and I implemented almost the same as you've described here, but using a poll-based stateful workflow model instead of queueing. The biggest challenge - which I haven't solved as seamlessly as I'd like - is supporting updates / deletes in the source. You don't seem to discuss it in this post, does Neum handle that?
We do support updates for some sources. Deletes not yet. For some sources we do polling which is then dumped on the queues. For other we have listeners that subscribe to changes.
What are the challenges you are facing in supporting this?
Are there any good implementations of using RAG within postgresql ecosystem? I have seen blogposts from supabase[0] and timescale db[1] but not a full fledged project. The full text search is very good within postgres at the moment and having semantic search within the same ecosystem is quiet helpful atleast for simple usecases. [0] https://supabase.com/docs/guides/database/extensions/pgvecto... [1] https://www.times…
Or generally what are good vector dbs have tried LlaMaindex, pinecone and milvus but all kinda sucked different way.
Good article BUT I can't fathom that people would use a managed service to generate and store embeddings. The openAI or replicate embeddings APIs are already a managed service... You would still need to self managing it all just into a different API. And dealing with embeddings is the kind of fun work every engineer wants to do anyway. Still a good article but very perplexing how the company can exist
IMO the fun parts are actually prototyping and figuring out the right pattern I want to use for my solution. Once you have done that, scaling and dealing with robustness tends to be a bit less fun.
Good article BUT I can't fathom that people would use a managed service to generate and store embeddings. The openAI or replicate embeddings APIs are already a managed service... You would still need to self managing it all just into a different API. And dealing with embeddings is the kind of fun work every engineer wants to do anyway. Still a good article but very perplexing how the company can exist
https://python.langchain.com/docs/modules/model_io/prompts/p...
Earlier quoted context omitted.
To some degree. The amount of data that will be brought into search solutions will be enormous, seems like a good time to try to reimagine what that process might look like
Also this is search for LLM not for humans so optimal solution will be different. Or even with models it is not that hard to imagine that Mistral-8b will need different results than GPT4 which has 1.76 trillion parameters.
But really I think that LLMs should use search as just one of their tools - just like humans do. I would call it Tool Augmented Generation. And also be able to reason through many hops. A good system answer the question _What is the 10th Fibonacci number?_ by looking up the definition in wikipedia, writing code for computing the sequence, testing and debugging it and executing it to compute the 10th number.
It seems to me that RAG is really search, and search is generally a hard problem without an easy one size fits all solution. E.g., as people push retrieval further and further in the context of LLM generation, they're going to go further down the rabbit hole of how to build a good search system. Is everyone currently reinventing search from first principles?
TL;DR: Queue upload events via SQS, upload files to s3, scale consumers based on queue length with keda and use haystack to turn files into embeddings.
This also works for arbitrary pipelines with your models, custom nodes (python code snippeds) and is pretty efficient.
Part1 (application&architecture): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Part2 (scaling): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Example code: https://github.com/ArzelaAscoIi/haystack-keda-indexing
We actually also stared with celery, but moved to SQS to improve the stability.
Earlier quoted context omitted.
we've been using pgvector at the 100M scale without any major problems so far, but I guess it depends on your specific use case. we've also been using elastic search dense vector fields which also seems to scale well, but of course its pricey but we already have it in our infra so works well.
What size are your embeddings?
Earlier quoted context omitted.
we've been using pgvector at the 100M scale without any major problems so far, but I guess it depends on your specific use case. we've also been using elastic search dense vector fields which also seems to scale well, but of course its pricey but we already have it in our infra so works well.
What type of latency requirements are you dealing with? (i.e. look up time, ingestion time) Were you using postgres already or migrated data into it?
For future lookup queries on pgvector, we can almost always pre-filter on an index before the vector search.
yes, we use postgres pretty extensively already.