Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database that can index 1B vectors in 48M

vectroid.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Vector database that can index 1B vectors in 48M

#61

Post by roflcopter69 »

nojvek wrote: post #60
roflcopter69 wrote: post #53

I think the whole field of vector databases is mostly just one huge misunderstanding. Most of you are not Google or any other big tech company so so won't have billions of embeddings. It's crazy how people add bloat and complexity to their stuff just because they want to do medium scale RAG with ca. 2 million embeddings. Here comes the punchline, you do not need a fancy vector database in this case. I stumbled over h…

There’s vss as duckdb extension too that builds a hnsw index. https://github.com/duckdb/duckdb-vss Since duckdb is already columnar, it goes brrrrr with single digit millisecond vector similarly lookups.

Oh right, duckdb being columnar is the ultimate brrr factor for such a brute force vector similarity search. But doesn't using a HNSW index completely forfeit this potential advantage?

Re: Vector database that can index 1B vectors in 48M

#62

Post by graphGL »

stevesimmons wrote: post #55
roflcopter69 wrote: post #53

I think the whole field of vector databases is mostly just one huge misunderstanding. Most of you are not Google or any other big tech company so so won't have billions of embeddings. It's crazy how people add bloat and complexity to their stuff just because they want to do medium scale RAG with ca. 2 million embeddings. Here comes the punchline, you do not need a fancy vector database in this case. I stumbled over h…

Might be all you need, except an open source licence: > For production or managed service use, please contact SQLite Cloud, Inc for a commercial license.

Try https://github.com/asg017/sqlite-vec - Apache 2, Mozilla backed!

Re: Vector database that can index 1B vectors in 48M

#63

Post by roflcopter69 »

graphGL wrote: post #62
stevesimmons wrote: post #55

Earlier quoted context omitted.

Might be all you need, except an open source licence: > For production or managed service use, please contact SQLite Cloud, Inc for a commercial license.

Try https://github.com/asg017/sqlite-vec - Apache 2, Mozilla backed!

I'd be cautious. Project seems abandoned. And I wouldn't say it's one of those cases where a piece of software is just finished and doesn't need any changes.

Re: Vector database that can index 1B vectors in 48M

#64

Post by roflcopter69 »

nojvek wrote: post #60
roflcopter69 wrote: post #53

I think the whole field of vector databases is mostly just one huge misunderstanding. Most of you are not Google or any other big tech company so so won't have billions of embeddings. It's crazy how people add bloat and complexity to their stuff just because they want to do medium scale RAG with ca. 2 million embeddings. Here comes the punchline, you do not need a fancy vector database in this case. I stumbled over h…

There’s vss as duckdb extension too that builds a hnsw index. https://github.com/duckdb/duckdb-vss Since duckdb is already columnar, it goes brrrrr with single digit millisecond vector similarly lookups.

Okay, bummer. No support for quantized datatypes yet and from the docs I cannot see anything that mentions fast brute force search. I personally don't need an index. But I see that https://github.com/unum-cloud/usearch which is used by duckdb-vss in turn uses https://github.com/ashvardanian/simsimd which should make a really fast exact vector similarity search possible. Am I missing something here?

Re: Vector database that can index 1B vectors in 48M

#65

Post by gaogao »

williamscales wrote: post #29
gaogao wrote: post #20

Earlier quoted context omitted.

Yeah, the SI symbol for minutes is min, if you're going to abbreviate it in a technical context. Super funky using M.

Agree the correct abbreviation is min. Nitpick: could be wrong but I don’t think minutes is an SI derived unit.

Yeah, it's under "Non-SI units accepted for use with SI", so more ISO than SI, but has standards for how it's used with SI units, esp since M has meaning there

Re: Vector database that can index 1B vectors in 48M

#66

Post by ashvardanian »

ab5tract wrote: post #59
ashvardanian wrote: post #56

Earlier quoted context omitted.

Lucene is tough to deal with. About 15 hours ago — right when this comment was posted — I was giving a talk at Databricks comparing the world’s most widely used search engines. I’ve never run into as many issues with any other similar tool as I did with Lucene. To be fair, it’s been around for ~26 years and has aged remarkably well... but it’s the last thing I’d choose today.

Can I ask you which alternatives exist at the layer Lucene occupies? I went looking around last year and couldn’t really find many options, but I might have been looking in the wrong places.

For Vector Search the top 2 are: Meta’s FAISS and (my) Unum’s USearch. Lucene powers Elastic, Solr, MongoDB Atlas, AWS OpenSearch, Azure Cognitive Search. USearch powers ClickHouse, DuckDB, YugaByte, TiDB, ScyllaDB, MemGraph, KuzuDB, Lantern, and a few big closed source names that don’t mention it, as far as I know. FAISS has the highest usage among Python developers, but if you are indexing large collections you should consider alternatives.
Post reply on HN