Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Vortex – a high-performance columnar file format

github.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Show HN: Vortex – a high-performance columnar file format

#51

Post by danking00 »

xiaodai wrote: post #42

There are a bunch of these including fst in the R ecosystem. JDF.jl in the julia ecosystem etc.

Thanks for introducing me to these other formats! I hadn't heard of them yet. All three of fst, JDF, and Vortex appear share the goal of high throughput (de)serialization of tabular data and random access to the data. However, it is not clear to me how JDF and fst permit random access on compressed data because both appear to use block compression (respectively Blosc and LZ4 or Zstd). While both Blosc and Zstd are extremely fast, accessing a single value of a single row necessarily requires decompressing a whole block of data. Instead of O(1) random access you get O(N_ROWS_PER_BLOCK) random access.

In Vortex, we've specifically invested in high throughput compression techniques that admit O(1) random access. These kinds of techniques are also sometimes called "lightweight compression". The DuckDB folks have a good writeup [1] on the common ones.

[1] https://duckdb.org/2022/10/28/lightweight-compression.html

Re: Show HN: Vortex – a high-performance columnar file format

#52

Post by agoose77 »

the_mitsuhiko wrote: post #3

> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…

For fun, the ROOT file format used in high energy physics has this kind of feature: https://root.cern.ch/root/SchemaEvolution.pdf

It's also a very old format, so not without its warts :)

Re: Show HN: Vortex – a high-performance columnar file format

#53

Post by CapeTheory »

speed_spread wrote: post #45
miere wrote: post #32

Imagine explaining to a newcomer that you write your app using Vert.x, it consumes AI models from GCP Vertex and uses Vortex for its high-performance columnar file structure.

You forgot mention of Verticle and Vectrex.

Also, the demo site is running on Vercel and the docs are written in LaTeX.

Re: Show HN: Vortex – a high-performance columnar file format

#55

Post by kwillets »

danking00 wrote: post #51
xiaodai wrote: post #42

There are a bunch of these including fst in the R ecosystem. JDF.jl in the julia ecosystem etc.

Thanks for introducing me to these other formats! I hadn't heard of them yet. All three of fst, JDF, and Vortex appear share the goal of high throughput (de)serialization of tabular data and random access to the data. However, it is not clear to me how JDF and fst permit random access on compressed data because both appear to use block compression (respectively Blosc and LZ4 or Zstd). While both Blosc and Zstd are ex…

This paper compares the benefits of lightweight compression and other techniques:

https://blog.acolyer.org/2018/09/26/the-design-and-implement...

Re: Show HN: Vortex – a high-performance columnar file format

#56

Post by amadio »

agoose77 wrote: post #52
the_mitsuhiko wrote: post #3

> One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification. That is quite interesting. One challenge in general with parqet and arrow in the otel / observability ecosystem is that the shape of data…

For fun, the ROOT file format used in high energy physics has this kind of feature: https://root.cern.ch/root/SchemaEvolution.pdf It's also a very old format, so not without its warts :)

There is also a new format being developed for Run 4, RNTuple:

- https://indico.fnal.gov/event/23628/contributions/240607/

- https://indico.cern.ch/event/1338689/contributions/6077632/

Re: Show HN: Vortex – a high-performance columnar file format

#57

Post by xiaodai »

danking00 wrote: post #51
xiaodai wrote: post #42

There are a bunch of these including fst in the R ecosystem. JDF.jl in the julia ecosystem etc.

Thanks for introducing me to these other formats! I hadn't heard of them yet. All three of fst, JDF, and Vortex appear share the goal of high throughput (de)serialization of tabular data and random access to the data. However, it is not clear to me how JDF and fst permit random access on compressed data because both appear to use block compression (respectively Blosc and LZ4 or Zstd). While both Blosc and Zstd are ex…

I see. Very nice. So it's a trade-off. I imagine the throughput of these light-weight compression suffers a little. In analytical workloads, it's common to do things like compute the mean of a vector or compute the gradient for this batch of data so random access appear less of an issue here.

Re: Show HN: Vortex – a high-performance columnar file format

#58

Post by danking00 »

xiaodai wrote: post #57
danking00 wrote: post #51

Earlier quoted context omitted.

Thanks for introducing me to these other formats! I hadn't heard of them yet. All three of fst, JDF, and Vortex appear share the goal of high throughput (de)serialization of tabular data and random access to the data. However, it is not clear to me how JDF and fst permit random access on compressed data because both appear to use block compression (respectively Blosc and LZ4 or Zstd). While both Blosc and Zstd are ex…

I see. Very nice. So it's a trade-off. I imagine the throughput of these light-weight compression suffers a little. In analytical workloads, it's common to do things like compute the mean of a vector or compute the gradient for this batch of data so random access appear less of an issue here.

We’ll post a blog post soon with specific, benchmarked numbers, but, in this case, you can have your cake and eat it too!

The compression and decompression throughputs of Vortex (and other lightweight compression schemes) are similar or better than Parquet for many common datasets. Unlike Zstd or Blosc, the lightweight encodings are, generally, both computationally simple and SIMD friendly. We’re seeing multiple gibibytes per second on an M2 MacBook Pro on various datasets in the PBI benchmark [1].

The key insight is that most data we all work with has common patterns that don’t require sophisticated, heavyweight compression algorithm. Let’s take advantage of that fact to free up more cycles for compute kernels!

[1] https://github.com/cwida/public_bi_benchmark

Re: Show HN: Vortex – a high-performance columnar file format

#59

Post by xiaodai »

danking00 wrote: post #58
xiaodai wrote: post #57

Earlier quoted context omitted.

I see. Very nice. So it's a trade-off. I imagine the throughput of these light-weight compression suffers a little. In analytical workloads, it's common to do things like compute the mean of a vector or compute the gradient for this batch of data so random access appear less of an issue here.

We’ll post a blog post soon with specific, benchmarked numbers, but, in this case, you can have your cake and eat it too! The compression and decompression throughputs of Vortex (and other lightweight compression schemes) are similar or better than Parquet for many common datasets. Unlike Zstd or Blosc, the lightweight encodings are, generally, both computationally simple and SIMD friendly. We’re seeing multiple gibi…

Cool looking forward to it.

Re: Show HN: Vortex – a high-performance columnar file format

#60

Post by cnity »

beAbU wrote: post #24
neeh0 wrote: post #6

Earlier quoted context omitted.

It gave me a moment of pause why Rust is part of the equation, but I concluded I'm too dumb

For a while "written in Rust" was sort of a "trust me, bro" label. The hivemind asserted that something written in rust must be automatically good and safe, because rust is good and safe. Thank god everyone wisened up. The tool maketh not the craftsman. These days the "written in rust" tag is met with knee jerk skepticism, as-if the hive mind over corrected.

Now the pendulum has swung too far in the opposite direction. The linked repo README makes no song and dance, barely even a mention (and even then just to explain some setup alongside Python instructions), and yet here we are obsessing over the source language of the repo to no benefit.
Post reply on HN