Earlier quoted context omitted.
Well you could store numbers all fine, but indexing vectors for similarity queries seems fairly recent and not all that widespread in the transactional world. As the traditional db move forward in the space the need for dedicated vector databases will likely shrink, except for some very specific implementation that offer unique enough features (I.e. deeplake does vector search over object storage, which is very conve…
sqlite has r-trees for instance [0]. Could it be good enough for most use cases? If it's to query a knowledge base for instance, a couple dimensions should be sufficient. With the added benefit of being able to query your data in other ways. [0] https://www.sqlite.org/rtree.html
Every database will become a vector database sooner or later
51–60 of 143 posts
Re: Every database will become a vector database sooner or later
#52Earlier quoted context omitted.
I mean NOSQL was hype with no substance but "you can scale more if you deal with not having ACID" is just generally true. Of course ACID scales to well into the Fortune 500 scale so...
"No substance" seems a bit harsh. They mostly seem a tarted up associative array, sure, but a key-value store is a thing.
Re: Every database will become a vector database sooner or later
#53Earlier quoted context omitted.
Exactly - vector indexes are so different than traditional RDBMS B-Tree or LSM Tree indexes that it doesn’t make sense to use the same store for both unless it’s basically a toy app. Someone makes the example in another comment, but it’s analogous to OLTP vs OLAP
I don't even want to imagine the workload on a high txn OLTP mixed with OLAP access pattern. IMHO If you can, you don't need OLAP in the first place.
Now you do NOT want to run such a setup on the same hardware that you use for your transactional systems, of course. But you CAN use the same software (like Oracle), which means that you do get some reduction in tech complexity.
Re: Every database will become a vector database sooner or later
#54I think the move towards vector databases might be more hype than necessity. Traditional databases, when properly optimized, can handle vector data for many use cases. The push for specialized vector databases could be re-evaluated in terms of efficiency and cost-effectiveness compared to optimizing existing scalar databases.
To be fair, Vector databases does sound more official as "new and important technology" compared to the last db hype of NOSQL.
NoSQL has been around for over 20+ years.
Since then Cassandra, DynamoDB, FoundationDB, MongoDB, Neo4J, Redis etc are not only still around but widely used and powering many of the services you use today.
Re: Every database will become a vector database sooner or later
#55Earlier quoted context omitted.
sqlite has r-trees for instance [0]. Could it be good enough for most use cases? If it's to query a knowledge base for instance, a couple dimensions should be sufficient. With the added benefit of being able to query your data in other ways. [0] https://www.sqlite.org/rtree.html
r*-trees work doesn't work well when the number of dimensions stored in the index is much higher than the logarithm of the number of indexed entries, and this is a prevailing property of divide-and-conquer spatial index types when the keyspace is divided based on a single dimension at a time. As vectors regularly have 100+ dimensions, normal spatial indexing methods applied to vectors wouldn't be very efficient for a…
Even if you could make it perform well, it would not do what you want.
Re: Every database will become a vector database sooner or later
#56Earlier quoted context omitted.
I mean NOSQL was hype with no substance but "you can scale more if you deal with not having ACID" is just generally true. Of course ACID scales to well into the Fortune 500 scale so...
"No substance" seems a bit harsh. They mostly seem a tarted up associative array, sure, but a key-value store is a thing.
DynamoDB underpins much of AWS which in turn underpins a ridiculous number of web services.
So definitely more than just a thing.
Re: Every database will become a vector database sooner or later
#57It is true that every major DB ventor, SQL or not, is smashing the AI/vector keyword on their front pages. In Elastic for example, their vector capabilities have gone from laughable to respectable in a year. Its a lot simpler to just use one DB instead of many. But a question for true DB experts here: 1. Is there any real advantage to building a dedicated vector DB from scratch? 2. Is vector DB something that can be…
Re: Every database will become a vector database sooner or later
#58This article is extremely correct and true, bordering on obvious. Vectors are a feature of a database engine that all engines will eventually offer -- not a new category of databases.
Would you say the same about graph databases? (e.g., Neo4j, ArangoDB, Neptune)
I bring this up because the ad hoc analytical use case for graph stores is so niche most engines haven't even seen enough demand to introduce it because you can always store graph relationships and offer retrievals in those engines to a limited depth which is typically sufficient for most operational / transactional use cases.
Re: Every database will become a vector database sooner or later
#59> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases. Co-locating vectors and original documents also reduces latency. Yet OLAP databases continue to thrive alongside OLTP databases, the nascence of NewSQL hybrid (HTAP) databases notwithstanding. Different needs dictate different design choices for optimality.
Is anyone considering a new OLAP system these days? If “NewSQL” (which seems to be a fancy buzzword for running analytics in your transactional database) takes off wont it be the final nail in the coffin for OLAP?
Re: Every database will become a vector database sooner or later
#60Supabase has pgvector extension and that’s enough for my limited RAG use cases. I dont really need to use anything beyond postgres. On the other hand, enterprise might find it easier/cheaper to buy a second db than migrating their existing db to whatever the latest version. I dont think it’s as simple
Exactly. We use Supabase too but are at a scale where it just made sense to use a second, dedicated vector db (Pinecone) than to bloat our Postgres db that has a completely different workload
I'd really love to know what kind of insane scale justifies that tradeoff...