41–50 of 62 posts
Two things I would hope to see before I'd start using Vorter is geospatial data support (there's already Geoparquet [1]) and WASM for in-browser Arrow processing. Things like Lonboard [2] and Observable framework [3] rely on Parquet, Arrow and Duckdb files for their powerful data analytics and visualisation.
Earlier quoted context omitted.
You're unlikely to find this with any analytic file format (including Vortex). The main reason is that OLAP systems generally assume an immutable distributed object/block layer (S3, HDFS, ABFS, etc.). It's then generally up to a higher-level component called a table format to handle the idea of edits. See for example how Apache Iceberg handles deletes https://iceberg.apache.org/spec/#row-level-deletes
This is true, and in principle a good thing, but in the time since Parquet and ORC were created GDPR and CCPA are things that have come to exist. Any format we build in that space, today, needs to support in-place record-level deletion.
> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…
Basically, since I ended up building a custom library for this, I wanted to solve the portability problem by making it stupidly simple to reverse engineer, so I cooked up a convention where each column (and supporting column) is a file, with a file name that describes its format and role.
So a real-world production table looks like this if you ls in the directory (omitting a few columns for brevity):
combinedId.0.dat.s64le.bin
documentMeta.0.dat.s64le.bin
features.0.dat.s32le.bin
size.0.dat.s32le.bin
termIds.0.dat-len.varint.bin
termIds.0.dat.s64le[].zstd
termMetadata.0.dat-len.varint.bin
termMetadata.0.dat.s8[].zstd
Imagine explaining to a newcomer that you write your app using Vert.x, it consumes AI models from GCP Vertex and uses Vortex for its high-performance columnar file structure.
> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…
I've been experimenting with taking this self-description paradigm even farther, for a file format I've cooked up for ephemeral data in my search engine. Basically, since I ended up building a custom library for this, I wanted to solve the portability problem by making it stupidly simple to reverse engineer, so I cooked up a convention where each column (and supporting column) is a file, with a file name that describ…
> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…
I've been experimenting with taking this self-description paradigm even farther, for a file format I've cooked up for ephemeral data in my search engine. Basically, since I ended up building a custom library for this, I wanted to solve the portability problem by making it stupidly simple to reverse engineer, so I cooked up a convention where each column (and supporting column) is a file, with a file name that describ…
Earlier quoted context omitted.
I've been experimenting with taking this self-description paradigm even farther, for a file format I've cooked up for ephemeral data in my search engine. Basically, since I ended up building a custom library for this, I wanted to solve the portability problem by making it stupidly simple to reverse engineer, so I cooked up a convention where each column (and supporting column) is a file, with a file name that describ…
Do you have a deeper writeup of this anywhere?
Can one edit it in place? That’s the main thing currently irritating me about parquet
You're unlikely to find this with any analytic file format (including Vortex). The main reason is that OLAP systems generally assume an immutable distributed object/block layer (S3, HDFS, ABFS, etc.). It's then generally up to a higher-level component called a table format to handle the idea of edits. See for example how Apache Iceberg handles deletes https://iceberg.apache.org/spec/#row-level-deletes