Show HN: Vortex – a high-performance columnar file format
1–10 of 62 posts
Re: Show HN: Vortex – a high-performance columnar file format
#2Not an expert in the space at all and it does seem like people are exploring new file and table formats so that is really cool!
How does this compare to Lance (https://lancedb.github.io/lance/)?
What do you think the key applied use case for Vortex is?
Re: Show HN: Vortex – a high-performance columnar file format
#3> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification.
That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data is not quite known with spans. There are arbitrary attributes on them, and they can change. To the best of my knowledge no particularly great solution exists today for encoding this. I wonder to which degree this system could be "abused" for that.
Re: Show HN: Vortex – a high-performance columnar file format
#4Thank God this file format is written in Rust, otherwise I'd be extremely skeptical.
Re: Show HN: Vortex – a high-performance columnar file format
#5Can one edit it in place?
That’s the main thing currently irritating me about parquet
Re: Show HN: Vortex – a high-performance columnar file format
#6Thank God this file format is written in Rust, otherwise I'd be extremely skeptical.
It gave me a moment of pause why Rust is part of the equation, but I concluded I'm too dumb
Re: Show HN: Vortex – a high-performance columnar file format
#7> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…
As someone who works in data schema on read formats like parquet are amazing. I hate having to guess schemas with CSVs.
Re: Show HN: Vortex – a high-performance columnar file format
#8Very cool! Any plans to offer more direct integrations with DataFusion, e.g. a `VertexReaderFactory`, hooks for pushdowns, etc?
Re: Show HN: Vortex – a high-performance columnar file format
#9“Vortex is a toolkit for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.”
So it’s a toolkit written in Rust. It is not a file format.
Re: Show HN: Vortex – a high-performance columnar file format
#10Very cool! Any plans to offer more direct integrations with DataFusion, e.g. a `VertexReaderFactory`, hooks for pushdowns, etc?
We have a TableProvider for use with Datafusion, checkout this crate and its examples: https://github.com/spiraldb/vortex/tree/develop/vortex-dataf...