Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database that can index 1B vectors in 48M

vectroid.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Vector database that can index 1B vectors in 48M

#41

Post by talipozturk »

f311a wrote: post #23
hungarianhc wrote: post #17

Earlier quoted context omitted.

Hey! It's a great question. Co-founder of Vectroid here. Today, the differences are going to be performance, price, accuracy, flexibility, and some intangible UI elegance. Performance: We actually INITIALLY built Vectroid for the use-case of billions of vectors and near single digit millisecond latency. During the process of building and talking to users, we found that there are just not that many use-cases (yet!) th…

I’m curious, what’s the tech stack behind this?

Vectroid is pure Java solution based on modified version of Lucene. We use a custom built FileSystem to work directly with GCS (Google cloud object store). It is a terraform/helm managed Kubernetes deployment.

Re: Vector database that can index 1B vectors in 48M

#42

Post by AlexClickHouse »

chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

USearch is this type of library: https://github.com/unum-cloud/usearch

Used in ClickHouse and a few other DBMS.

Re: Vector database that can index 1B vectors in 48M

#43

Post by chatmasta »

talipozturk wrote: post #40
chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

I think we have so many of those nice open source libraries but the problem is not the library or the algorithm (hsnw or ivf derivatives).. the problem is figuring out the right distributed architecture to balance cost, accuracy (recall) and speed (latency). I believe no single library will give you all that. For instance if you don't separate writes (indexing) from reads (queries) and scale them separately then your…

Yeah, I feel like these libraries are all one level lower than what I’m asking for. We need something that makes more assumptions (e.g. “I’m running as a component of some kind of database”) but… makes less decisions? Is more flexible? Idk. This is the hard part.

DataFusion nailed this balance between an embedded query engine and a standalone database system. It brings just the right amount of batteries that it’s not a super generic thing that does nothing useful out of the box, but it doesn’t bring so many that it needs to compete with full database systems.

I believe the maintainers refer to it as “the IR of databases” and I’ve always liked that analogy. That’s what I’d like to see for vector engines.

Maybe what we need as a pre-requisite is the equivalent of arrow/parquet ecosystem for vectors. DataFusion really leverages those standards for interoperability and performance. This also goes a long way toward the architectural decisions you reference — Arrow and Parquet are a solid, “good enough” choice for in-memory and storage formats that are efficient and flexible and well-supported. Is there something similar for vector storage?

Re: Vector database that can index 1B vectors in 48M

#44

Post by talipozturk »

ashvardanian wrote: post #30
talipozturk wrote: post #21

Earlier quoted context omitted.

No special hardware. Google Cloud vms. We use multiple of them during index building.

The question is how many, and what kind of VMs you use? It greatly affects performance :) I run a lot of search-related benchmarks ( https://github.com/ashvardanian ) and curious if you’ve compared to other engines on the same hardware setup, tracing recall, NDCG, indexing, and query speeds.

We shard the data and index on about 6 x n2-standard-96 spot instances so the total cost of indexing the entire deep1b is less than $12. We are working on to make it $6. We separate indexing and query VMs. For queries we use dedicated VMs. USearch numbers look great and are better than ours if you run the query and indexing on the same VM/node. We believe design-wise distributed, task-oriented design is the right way to handle vector search for thousands of tenants with different size datasets. Data ingest is also a separate task for us so Ingest, Index and Query are all handled by different cluster of VMs.

Re: Vector database that can index 1B vectors in 48M

#45

Post by jeadie »

chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

We’re building vector indexes into Datafusion for search (starting with S3 vectors).

Open source at https://github.com/spiceai/spiceai

Re: Vector database that can index 1B vectors in 48M

#46

Post by maxxen »

chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

Soo… usearch? Its literally one header file (of what use to be strict c++11). Funnily enough that is what is used in the official duckdb-vss extension.

Disclaimer: I wrote duckdb-vss

Re: Vector database that can index 1B vectors in 48M

#47

Post by whakim »

talipozturk wrote: post #40
chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

I think we have so many of those nice open source libraries but the problem is not the library or the algorithm (hsnw or ivf derivatives).. the problem is figuring out the right distributed architecture to balance cost, accuracy (recall) and speed (latency). I believe no single library will give you all that. For instance if you don't separate writes (indexing) from reads (queries) and scale them separately then your…

I couldn't agree with this more. I don't think the majority of problems with vector search at scale are vector search problems (although filtering + ANN is definitely interesting), they're search-problems-at-scale problems.

Re: Vector database that can index 1B vectors in 48M

#48

Post by itake »

chatmasta wrote: post #38

I would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every…

why not use this? https://github.com/facebookresearch/faiss

Re: Vector database that can index 1B vectors in 48M

#49

Post by talipozturk »

cluckindan wrote: post #26

How is this different from running tuned HNSW vector indices on Elasticsearch?

co-founder of Vectroid: We forked Lucene. Lucene is awesome for search in general, filters, and obviously full-text search. Very mature and well supported by so many big names and amazing engineers. So we take advantage of that but we had to change a few things to make it work perfectly for Vector use-case. We basically think Vector should be the main data type as it is the most difficult one to deal with. For instance, we modified Lucene to use X number of CPU / threads to build a single segment index. As a result, if/when needed, we can utilize hundreds of CPUs to index quicker and generate less number of segments that will enable lower query latency. We also built a custom File System Directory for Lucene to work off of GCS directly (or S3 later on). It can by-pass the kernel, read from network and write directly into the memory... no SSD, no page-cache, no mmap involved. Perhaps I should not say more...

Re: Vector database that can index 1B vectors in 48M

#50

Post by kingstnap »

adastra22 wrote: post #32
1999-03-31 wrote: post #18

1B vectors is nothing. You don’t need to index them. You can hold them in VRAM on a single node and run queries with perfect accuracy in milliseconds

1B x 4096 = 4T scalars. That doesn't fit in anyone's video ram.

Well we have AI GPUs now so you could do it.

Each MI325x has 256 GB of HBM. So you would need ~32 of em if it was 2 bytes per scalar.

Post reply on HN