51–60 of 61 posts
or you could just use postgres + pgvector? which many apps already have installed by default.
If one had instead 1M items, the situation would be completely different.
I still don't like dataframes but oh my God Polars is so much better than pandas. I was doing some time series calculations, simple equity price adjustments basically, in Polars and my two thoughts were: - WTF, I can actually read the code and test it. - it's running so fast it seems like it's broken.
Polars makes programming fun again instead of a chore.
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
> The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. parquet is columnar storage, so it’s use case is lots of heavy filtering/aggregation within analytical workloads (OLAP). consistent writes / updates, i.e. basically transactional (OLTP), use cases are never going to have great performance in columnar storage. its the wrong format to use for that. for faster w…
When you run a read query, there's one phase that determines the offsets where values are stored and another that reads the value at a given offset. For an update query that doesn't change the offsets, you can change the direction from reading the value at an offset to writing a new value to that location instead, and it should be plenty fast.
Parquet libraries just don't seem to consider that use case worth supporting for some reason and expect people to generate an entire new file with mostly the same content instead. Which definitely doesn't have great performance!
Earlier quoted context omitted.
> The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. parquet is columnar storage, so it’s use case is lots of heavy filtering/aggregation within analytical workloads (OLAP). consistent writes / updates, i.e. basically transactional (OLTP), use cases are never going to have great performance in columnar storage. its the wrong format to use for that. for faster w…
There's no reason why an update query that doesn't change the file layout and only twiddles some values in place couldn't be made fast with columnar storage. When you run a read query, there's one phase that determines the offsets where values are stored and another that reads the value at a given offset. For an update query that doesn't change the offsets, you can change the direction from reading the value at an of…
Also, many applications that require updates also update conditionally (update a where b = c). This requires re-synthesizing (at least some of) the row to make a comparison, another relatively expensive operation for a column store.
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
> The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. You can use glob patterns in DuckDB to query remote parquets though to get around this? Maybe break things up using a hive partitioning scheme or similar.
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
Yes, it is not useful for continuous writes and updates, but it is not what it is designed for. Use a database (e.g. SQLite just like you suggested) if you want to ingest real time/streaming data.
The trouble with Parquet (and columnar storage) in ML is, 1. You don't really care too-much about accessing subsets of columns 2. You can't easily append stuff to closed Parquet files. 3. Batched-row access is presumably slower due to lower cache-hits. It's okay for map-reduce style stuff where this doesn't matter, but in ML these limitations are an annoyance. HDF5 (or Zarr, less portably) solves some/many of these i…
Also batched-row access is negligible given the compression benefits you get with the columnar format, which is probably why it's still king in ML; I think given what I'm seeing in the industry and recent trends (e.g. Velox).
The problem with Parquet is it’s static. Not good for use cases that involve continuous writes and updates. Although I have had good results with DuckDB and Parquet files in object storage. Fast load times. If you host your own embedding model, then you can transmit numpy float32 compressed arrays as bytes, then decode back into numpy arrays. Personally I prefer using SQLite with usearch extension. Binary vectors the…
Earlier quoted context omitted.
There's no reason why an update query that doesn't change the file layout and only twiddles some values in place couldn't be made fast with columnar storage. When you run a read query, there's one phase that determines the offsets where values are stored and another that reads the value at a given offset. For an update query that doesn't change the offsets, you can change the direction from reading the value at an of…
Columnar storage systems rarely store the raw value at fixed position. They store values as run length encoded, dictionary encoded, delta encoded, etc... and then store metadata about chunk of values for pruning at query time. So rarely can you seek to an offset and update a value. The compression achieved means less data to read from disk when doing large scans and lower storage costs for very-large-datasets that ar…
But it's... Fine? Batch writes and rewrite dirty parts. Most of our cases are either appending events, or enriching with new columns, which can be modeled columnarly. It is a bit more painful in GPU land bc we like big chunks (250MB-1GB) for saturating reads, but CPU land is generally fine for us.
We have been eyeing iceberg and friends as a way to automate that, so I've been curious how much of the optimization, if any, they take for us
Lots of great findings --- I'm curious if anyone knows whether it is better to pass structured data or unstructured data to embedding api's? If I ask ChatGPT, it says it is better to send unstructured data. (looking at the authors github, it looks like he generated embeddings from json strings) My use case is for jsonresume, I am creating embeddings by sending full json versions as strings, but I've been experimentin…
In general I like to send structured data (see the input format here: https://github.com/minimaxir/mtg-embeddings ), but the ModernBERT base for the embedding model used here specifically has better benefits implicitly for structured data compared to previous models. That's worth another blog post explaining why.