Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every database will become a vector database sooner or later

nextword.substack.com

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Every database will become a vector database sooner or later

#41

Post by crazy_marksman »

weird-eye-issue wrote: post #19
muratsu wrote: post #3

Supabase has pgvector extension and that’s enough for my limited RAG use cases. I dont really need to use anything beyond postgres. On the other hand, enterprise might find it easier/cheaper to buy a second db than migrating their existing db to whatever the latest version. I dont think it’s as simple

Exactly. We use Supabase too but are at a scale where it just made sense to use a second, dedicated vector db (Pinecone) than to bloat our Postgres db that has a completely different workload

How do you deal with security and access control across postgres and pinecone?

Re: Every database will become a vector database sooner or later

#42

Post by andre-z »

BoorishBears wrote: post #35
andre-z wrote: post #31

[flagged]

Super lazy comment. SQL databases got JSON handling. NoSQL databases added indexes and ACID compliance. An entire new class of databases that operate identically to existing ones with the exception of a single column type is silly, it's only getting traction because of aggressive VC-funded marketing.

Would you say the same about Keyword Search engines like Elastic, Solr, etc? It is just another column type, full-text index, that is available in any proper database. Just a hype...

Re: Every database will become a vector database sooner or later

#43

Post by thesz »

avereveard wrote: post #17
loondri wrote: post #14

I think the move towards vector databases might be more hype than necessity. Traditional databases, when properly optimized, can handle vector data for many use cases. The push for specialized vector databases could be re-evaluated in terms of efficiency and cost-effectiveness compared to optimizing existing scalar databases.

Well you could store numbers all fine, but indexing vectors for similarity queries seems fairly recent and not all that widespread in the transactional world. As the traditional db move forward in the space the need for dedicated vector databases will likely shrink, except for some very specific implementation that offer unique enough features (I.e. deeplake does vector search over object storage, which is very conve…

What is "vector search over object storage?" Does deeplake performs some computations on objects and search on their embeddings?

Re: Every database will become a vector database sooner or later

#44

Post by panda888888 »

mr_toad wrote: post #23
esafak wrote: post #4

> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases. Co-locating vectors and original documents also reduces latency. Yet OLAP databases continue to thrive alongside OLTP databases, the nascence of NewSQL hybrid (HTAP) databases notwithstanding. Different needs dictate different design choices for optimality.

Is anyone considering a new OLAP system these days? If “NewSQL” (which seems to be a fancy buzzword for running analytics in your transactional database) takes off wont it be the final nail in the coffin for OLAP?

I think the opposite concept, "reverse ETL," is actually more popular. You put everything into your data warehouse and then pump whatever you need out from there.

Re: Every database will become a vector database sooner or later

#45

Post by nhourcard »

esafak wrote: post #4

> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases. Co-locating vectors and original documents also reduces latency. Yet OLAP databases continue to thrive alongside OLTP databases, the nascence of NewSQL hybrid (HTAP) databases notwithstanding. Different needs dictate different design choices for optimality.

Different needs dictate different design choices for optimality.

Could not agree more. Even for time series, which could be seen as a subset of OLAP, trade-offs and design choices inherent to time-series data are necessary. As an example of a TSDB that I know well, QuestDB: Data is always ordered by time once it lands on the disk, the data is partitioned by time, and the ingestion protocol is conceived to stream large volumes of data, which can be either continuous or in bursts.

Re: Every database will become a vector database sooner or later

#47

Post by BoorishBears »

andre-z wrote: post #42
BoorishBears wrote: post #35

Earlier quoted context omitted.

Super lazy comment. SQL databases got JSON handling. NoSQL databases added indexes and ACID compliance. An entire new class of databases that operate identically to existing ones with the exception of a single column type is silly, it's only getting traction because of aggressive VC-funded marketing.

Would you say the same about Keyword Search engines like Elastic, Solr, etc? It is just another column type, full-text index, that is available in any proper database. Just a hype...

Look up what Lucene is and how it differs from what an RDBMS offers for full-text search and you'll be able to answer that question for yourself.

I had a feeling the heavily overdone sarcasm in the original comment was a way to mask a lack of understanding of the subject, and here you confirmed it.

_

And maybe call out the fact you're the founder at a vector database SaaS? Definitely explains why you'd put out that original diatribe...

Re: Every database will become a vector database sooner or later

#48

Post by smitty1e »

Guvante wrote: post #38
mnky9800n wrote: post #34

Earlier quoted context omitted.

To be fair, Vector databases does sound more official as "new and important technology" compared to the last db hype of NOSQL.

I mean NOSQL was hype with no substance but "you can scale more if you deal with not having ACID" is just generally true. Of course ACID scales to well into the Fortune 500 scale so...

"No substance" seems a bit harsh.

They mostly seem a tarted up associative array, sure, but a key-value store is a thing.

Re: Every database will become a vector database sooner or later

#49

Post by [deleted] »

Guvante wrote: post #38
mnky9800n wrote: post #34

Earlier quoted context omitted.

To be fair, Vector databases does sound more official as "new and important technology" compared to the last db hype of NOSQL.

I mean NOSQL was hype with no substance but "you can scale more if you deal with not having ACID" is just generally true. Of course ACID scales to well into the Fortune 500 scale so...

[deleted]
Post reply on HN