41–50 of 61 posts
I still don't like dataframes but oh my God Polars is so much better than pandas. I was doing some time series calculations, simple equity price adjustments basically, in Polars and my two thoughts were: - WTF, I can actually read the code and test it. - it's running so fast it seems like it's broken.
Really cool article, I've enjoyed your work for a long time. You might add a note for those jumping into a sqlite implementation, that duckdb reads parquet and launched a few vector similarity functions which cover this use-case perfectly: https://duckdb.org/2024/05/03/vector-similarity-search-vss.h...
I have tinkered with using DuckDB as a poor man's vector database for a POC and had great results. One thing I'd love to see is being able to do some sort of row group level metadata statistics for embeddings within a parquet file - something that would allow various readers to push predicates down to an HTTP request metadata level and completely avoid loading in non-relevant rows to the database from a remote file -…
https://github.com/jasonjmcghee/portable-hnsw
Opens up efficient query patterns for larger datasets for RAG projects where you may not have the resources to run an expensive vector database
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
You can use glob patterns in DuckDB to query remote parquets though to get around this? Maybe break things up using a hive partitioning scheme or similar.
I still don't like dataframes but oh my God Polars is so much better than pandas. I was doing some time series calculations, simple equity price adjustments basically, in Polars and my two thoughts were: - WTF, I can actually read the code and test it. - it's running so fast it seems like it's broken.
There’s some nice plugins too, some are finance related: https://github.com/ddotta/awesome-polars
Earlier quoted context omitted.
I have tinkered with using DuckDB as a poor man's vector database for a POC and had great results. One thing I'd love to see is being able to do some sort of row group level metadata statistics for embeddings within a parquet file - something that would allow various readers to push predicates down to an HTTP request metadata level and completely avoid loading in non-relevant rows to the database from a remote file -…
I thought about this some more and did some research - and found an indexing approach using HNSW, serialized to parquet, and queried from the browser here: https://github.com/jasonjmcghee/portable-hnsw Opens up efficient query patterns for larger datasets for RAG projects where you may not have the resources to run an expensive vector database
As others have mentioned in other threads, parquet isn't a great tool for the job here, but you could theoretically build a different file format that lends itself better to the problem of static file(s) representing a vector database.
Earlier quoted context omitted.
There’s some nice plugins too, some are finance related: https://github.com/ddotta/awesome-polars
The one thing I really want is for someone to make it so I can use it in F#. Presumably it's possible given how the python bit is implemented under the hood?
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
parquet is columnar storage, so it’s use case is lots of heavy filtering/aggregation within analytical workloads (OLAP).
consistent writes / updates, i.e. basically transactional (OLTP), use cases are never going to have great performance in columnar storage. its the wrong format to use for that.
for faster writes/updates you’d want row-based, i.e. CSV or an actual database. which i’m glad to see is where you kind of ended up anyway.
This trick can be used to reduce your payload sizes. In Vespa, they support this format which is particularly useful when the same vectors are referenced multiple times in a document. For ColBERT or ColPaLi like cases (where you have many embedding vectors), this can reduce the size of the vectors stored on disk massively.
https://docs.vespa.ai/en/reference/document-json-format.html...
Not sure why this is not more commonly adopted though
Earlier quoted context omitted.
That would get around memory limitations but I still think that would be slow.
You'd be surprised. As long as your query is using Polars natives and not a UDF (which drops it down to Python), you may get good results.
The trouble with Parquet (and columnar storage) in ML is, 1. You don't really care too-much about accessing subsets of columns 2. You can't easily append stuff to closed Parquet files. 3. Batched-row access is presumably slower due to lower cache-hits. It's okay for map-reduce style stuff where this doesn't matter, but in ML these limitations are an annoyance. HDF5 (or Zarr, less portably) solves some/many of these i…
The case of 1. really depends on the workload. For embeddings etc selecting column subsets is rare. In order cases, where one has a a bunch of separate features, doing column subsetting might be rather common. But yes, it is far from every case.