The best way to use text embeddings portably is with Parquet and Polars
1–10 of 61 posts
Re: The best way to use text embeddings portably is with Parquet and Polars
#2Since we are talking about an embedded solution shouldn't the benchmark be something like sqlite with a vector extension or lancedb?
Re: The best way to use text embeddings portably is with Parquet and Polars
#3Since we are talking about an embedded solution shouldn't the benchmark be something like sqlite with a vector extension or lancedb?
I mention sqlite + sqlite-vec at the end, noting it requires technical overhead and it's not as easy as read_parquet() and write_parquet().
I just became aware of lancedb and am looking into that, although from glancing at the README it has similar issues to faiss with regards to usability for casual use, although much better than faiss in that it can work with colocated metadata.
Re: The best way to use text embeddings portably is with Parquet and Polars
#4This is pretty neat.
IMO a hindrance to this was lack of built-in fixed-size list array support in the Arrow format, until recently. Some implementations/clients supported it, while others didn't. Else, it could have been used as the default storage format for numpy arrays, torch tensors, too.
(You could always store arrays as variable length list arrays with fixed strides and handle the conversion).
Re: The best way to use text embeddings portably is with Parquet and Polars
#5Is your example of a float32 number correct, holding 24 ascii char representation? I had thought single-precision gonna be 7 digits and the exponent, sign and exp sign. Something like 7+2+1+1 or 10 char ascii representation? Rather than the 24 you mentioned?
Re: The best way to use text embeddings portably is with Parquet and Polars
#6Is your example of a float32 number correct, holding 24 ascii char representation? I had thought single-precision gonna be 7 digits and the exponent, sign and exp sign. Something like 7+2+1+1 or 10 char ascii representation? Rather than the 24 you mentioned?
It depends on the default print format. The example string I mentioned is pulled from what np.savetxt() does (fmt='%.18e') and there isn't any precision loss in that number. But I admit I'm not a sprintf() guru.
In practice numbers with that much precision is overkill and verbose so tools don't print float32s to that level of precision.
Re: The best way to use text embeddings portably is with Parquet and Polars
#7Really cool article, I've enjoyed your work for a long time. You might add a note for those jumping into a sqlite implementation, that duckdb reads parquet and launched a few vector similarity functions which cover this use-case perfectly:
https://duckdb.org/2024/05/03/vector-similarity-search-vss.h...
Re: The best way to use text embeddings portably is with Parquet and Polars
#8For another library that has great performance and features like full text indexing and the ability to version changes I’d recommend lancedb https://lancedb.github.io/lancedb/
Yes, it’s a vector database and has more complexity. But you can use it without creating indexes and it has excellent polars and pandas zero copy arrow support also.
Re: The best way to use text embeddings portably is with Parquet and Polars
#9Check out Unum’s usearch. It beats anything, and is super easy to use. It just does exactly what you need.
Re: The best way to use text embeddings portably is with Parquet and Polars
#10For another library that has great performance and features like full text indexing and the ability to version changes I’d recommend lancedb https://lancedb.github.io/lancedb/ Yes, it’s a vector database and has more complexity. But you can use it without creating indexes and it has excellent polars and pandas zero copy arrow support also.
Lance is made for this stuff; parquet is not.