Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every database will become a vector database sooner or later

nextword.substack.com

Post reply on HN
Print view View original

131–140 of 143 posts

Re: Every database will become a vector database sooner or later

#131

Post by geoduck14 »

ChrisPzilla wrote: post #120

I think the only thing that can tell us whether or not every database can be a vector database is time. We have seen this with Time Series databases that they are their own unique type of database, and I believe this will be the same with Vector Databases.

>I think the only thing that can tell us whether or not every database can be a vector database is time.

I can help you put right now. You don't even need a database to have a "vector database". FAISS is an in-memory "vector database" that runs on the data that you happen to have. So: if you data is stored in a .txt file, load it into memory, index it with FAISS... bam a vector database from a data file.

Can we take X arbitrary "real" database and implement KNN search on top of 1000 indexed columns, I'm sure it is possible - but I'm also pretty sure most databases will die under the pressure (source: I've asked some of my favorite DBAs if I could do this and they said "no")

Re: Every database will become a vector database sooner or later

#132

Post by samlambert »

totetsu wrote: post #28
shri_krishna wrote: post #8

The one DB fits all approach only works when the size of the database is really small and never grows. Imagine you have 100 customers. Each customer generates, on average, a million 1536 dimension vector embeddings (considering OpenAI Ada dimensions which is the most popular right now). That is 6GB (1536 x 4 bytes per dimension for f32 x 1000_000) of just embeddings PER CUSTOMER. If you use HNSW it will take at least…

Are there any DB that could support both use cases while being able to partition them in such a way that the transactions etc are only kept on part of the resources they need to be. Basically two seperated DBs but sharing the same interfaces and security etc.

We (PlanetScale) announced Vector storage and search today. If I am understanding your request it sounds like something could do. I would love to hear more if you are willing to chat? s@planetscale.com

Re: Every database will become a vector database sooner or later

#133

Post by weird-eye-issue »

BoorishBears wrote: post #122
weird-eye-issue wrote: post #94

Earlier quoted context omitted.

That's a very exaggerated way to look at things lol. Nothing got bloated at all in this process, we are just using the right tools for the job. I'm a solo founder and the only backend developer. I can assure you this decision only made my life easier by choosing the correct tech from the get go.

Nothing exaggerated, your comment implied scale was your justification: in which case there'd better have been some crazy high load that just brought the tool you already had to its knees to justify paying an additional closed source platform and manually having to pipe data to it in addition to your main data store. Of course if I sounded incredulous it's because I didn't think you had that scale, and it sounds like…

No, you're not correct. We have over $2M ARR and with the amount of data we are storing it would be downright stupid to use Supabase.

We don't also "pipe" our data to Supabase, we use a couple different data stores depending on the best use case. For example we also use R2 and Durable Objects.

Just because you have a hammer doesn't mean everything is a nail.

Re: Every database will become a vector database sooner or later

#134

Post by gk1 »

redwood wrote: post #129
gk1 wrote: post #126

Earlier quoted context omitted.

Yes, for example any site with user-uploaded content — think marketplaces, social media, SaaS file/docs storage, etc. Very high write throughput and expectation of that data being available in search/chatbots right away.

But in a sense doesn't that lend credence to the idea that the closer the indexing is happening to where the data is born, there may be some natural advantages?

The “force merge” process in an Elastic vector index took as long as 18 hours in one test. That’s a lot longer than a few hops across the network.

This is not even a dig at Elastic. The problem is deeper than that… It’s an issue with the underlying vector index they (and many others) chose to bolt on, HNSW, which was not designed with frequent live updates in mind.

We have a post coming soon that covers the technical parts of this in more detail. You asked a good question.

Re: Every database will become a vector database sooner or later

#135

Post by redwood »

gk1 wrote: post #134
redwood wrote: post #129

Earlier quoted context omitted.

But in a sense doesn't that lend credence to the idea that the closer the indexing is happening to where the data is born, there may be some natural advantages?

The “force merge” process in an Elastic vector index took as long as 18 hours in one test. That’s a lot longer than a few hops across the network. This is not even a dig at Elastic. The problem is deeper than that… It’s an issue with the underlying vector index they (and many others) chose to bolt on, HNSW, which was not designed with frequent live updates in mind. We have a post coming soon that covers the technical…

Got it (now you've got me going down the rabbit hole here reading things like https://stackoverflow.com/questions/60226215/why-segment-mer...). I think the key question is whether a healthy index in operation with a healthy amount of resources and configuration should ever require a force merge or if that's effectively an anti-pattern that suggests you're already up a cul-de-sac of risk and from there it sounds like if you have any segments larger than 5 GB in particular you may find yourself stuck unless you're an expert.

I'd love to see someone who has some expert knowledge of elastic chime in to hear if the characterization here seems right? But admittedly not everyone's going to be a power user so if this isn't easy there's definitely a problem.

It definitely makes sense that the faster the rate of change the more likely the engine has to combine results from multiple places until optimal placement on disc is found in steady state later, and hence query latency would rise.

Re: Every database will become a vector database sooner or later

#136

Post by eggy »

hoosieree wrote: post #100
nwsm wrote: post #97

Earlier quoted context omitted.

It looks like array/stack Brainfuck. Previous uiua discussion: https://news.ycombinator.com/item?id=37673127 Brainfuck: https://en.wikipedia.org/wiki/Brainfuck

Appearances can be deceiving. In terms of expressiveness, Uiua is to Brainfuck as Python is to nand.

Exactly this. J and Uiua along with APL for me are very expressive. It doesn't take long to become comfortable with the glyphs. Sad to say, BQN's approach resonated with me, however, some subjective bias in me, didn't feel warm and fuzzy about the glyphs. And that's saying a lot, since I am good with J's ASCII noise, APL's glyphs, and now Uiua's glyphs. Sorry, Marshall! BQN is incredible. Maybe I had some sort of PTSD from my days at the Brooklyn Museum and some weird confluence of hieroglyphics and BQN glyphs!

Re: Every database will become a vector database sooner or later

#138

Post by BoorishBears »

weird-eye-issue wrote: post #133
BoorishBears wrote: post #122

Earlier quoted context omitted.

Nothing exaggerated, your comment implied scale was your justification: in which case there'd better have been some crazy high load that just brought the tool you already had to its knees to justify paying an additional closed source platform and manually having to pipe data to it in addition to your main data store. Of course if I sounded incredulous it's because I didn't think you had that scale, and it sounds like…

No, you're not correct. We have over $2M ARR and with the amount of data we are storing it would be downright stupid to use Supabase. We don't also "pipe" our data to Supabase, we use a couple different data stores depending on the best use case. For example we also use R2 and Durable Objects. Just because you have a hammer doesn't mean everything is a nail.

Well maybe we have different definitions of scale: I think my team spends about $2M a month on compute, so we don't pride ourselves on randomly pulling in new vendors.

Re: Every database will become a vector database sooner or later

#139

Post by weird-eye-issue »

BoorishBears wrote: post #138
weird-eye-issue wrote: post #133

Earlier quoted context omitted.

No, you're not correct. We have over $2M ARR and with the amount of data we are storing it would be downright stupid to use Supabase. We don't also "pipe" our data to Supabase, we use a couple different data stores depending on the best use case. For example we also use R2 and Durable Objects. Just because you have a hammer doesn't mean everything is a nail.

Well maybe we have different definitions of scale: I think my team spends about $2M a month on compute, so we don't pride ourselves on randomly pulling in new vendors.

You are incredibly dense

Re: Every database will become a vector database sooner or later

#140

Post by BoorishBears »

weird-eye-issue wrote: post #139
BoorishBears wrote: post #138

Earlier quoted context omitted.

Well maybe we have different definitions of scale: I think my team spends about $2M a month on compute, so we don't pride ourselves on randomly pulling in new vendors.

You are incredibly dense

When you're playing checkers people playing by the rules of chess might seem dense.
Post reply on HN