I recently had to solve the Entity Resolution Problem at my workplace, and here was how i went about it. Problem Statement: - Find the right entity for a query among ~50m entities. - Queries could be few of the entity attributes (If entities have n attributes, query can in 1..n) - Queries can mention attributes in various ways (Partial information, Typing errors, abbreviation, Extra information etc) Existing Solution…
Can you add data to your solution in realtime and is it possible to delete records? How and where are you hosting your solution (on premise, cloud, ....) is it highly available and does it automatically scale? Did you try zentity (for elastic) or zingg.ai?
It's supports all CRUD operations, used pinecone as the vector storage and it's highly available and can be scaled to additional pods, existing solution was using zentity.