This is such a leap forward it's hard to believe it's anything but magic.
Compared to columnar databases this is more of an incremental improvement.
OpenZL: An open source format-aware compression framework
11–20 of 110 posts
Re: OpenZL: An open source format-aware compression framework
#12In addition to the blog post, here are the other things we've published today: Code: https://github.com/facebook/openzl Documentation: https://openzl.org/ White Paper: https://arxiv.org/abs/2510.03203
We'll put those links in the toptext above.
Re: OpenZL: An open source format-aware compression framework
#13How do you use it to compress a directory (or .tar file)? Not seeing any example usages in the repo, `zli compress -o dir.tar.zl dir.tar` ->
Invalid argument(s):
No compressor profile or serialized compressor specified.
Edit: @terrelln Got it, thank you!
Re: OpenZL: An open source format-aware compression framework
#14Is this similar to Basis ? https://github.com/BinomialLLC/basis_universal
Re: OpenZL: An open source format-aware compression framework
#15How do you use it to compress a directory (or .tar file)? Not seeing any example usages in the repo, `zli compress -o dir.tar.zl dir.tar` -> Invalid argument(s): No compressor profile or serialized compressor specified. Same thing for the `train` command. Edit: @terrelln Got it, thank you!
There's a Quick Start guide here:
https://openzl.org/getting-started/quick-start/
However, OpenZL is different in that you need to tell the compressor how to compress your data. The CLI tool has a few builtin "profiles" which you can specify with the `--profile` argument. E.g. csv, parquet, or le-u64. They can be listed with `./zli list-profiles`.
You can always use the `serial` profile, but because you haven't told OpenZL anything about your data, it will just use Zstandard under the hood. Training can learn a compressor, but it won't be able to learn a format like `.tar` today.
If you have raw numeric data you want to throw at it, or Parquets or large CSV files, thats where I would expect OpenZL to perform really well.
Re: OpenZL: An open source format-aware compression framework
#16Is this similar to Basis ? https://github.com/BinomialLLC/basis_universal
No, not really. They are both cool but solve different problems. The problem Basis solves is that GPUs don't agree on which compressed texture formats to support in hardware. Basis is a single compressed format that can be transcoded to almost any of the formats GPUs support, which is faster and higher quality than e.g. decoding a JPEG and then re-encoding to a GPU format.
Re: OpenZL: An open source format-aware compression framework
#17It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827
Author of [0] here. Congratulations and well done for resisting. Eager to try it!
Edit: Have you any specific advice for training a fasta compressor beyond that given in e.g. "Using OpenZL" (https://openzl.org/getting-started/using-openzl/)
Re: OpenZL: An open source format-aware compression framework
#18It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827
That post immediately came to my mind too! Do you maybe have a comparison to share with respect to the specialized compressor mentioned in the OP there?
> Grace Blackwell’s 2.6Tbp 661k dataset is a classic choice for benchmarking methods in microbial genomics. (...) Karel Břinda’s specialist MiniPhy approach takes this dataset from 2.46TiB to just 27GiB (CR: 91) by clustering and compressing similar genomes together.
Re: OpenZL: An open source format-aware compression framework
#19Well, well. Kind of surprised to see this really good tool that should have been made available a longer time ago since the approach is quite sound.
When the data container is understood, the deduplication is far more efficient because now it is targeted.
Licensed as BSD-3-Clause, solid C++ implementation, well documented.
Will be looking forward to see new developments as more file formats are contributed.
Re: OpenZL: An open source format-aware compression framework
#20So, as I understand, you describe the structure of your data in an SDL and then the compressor can plan a strategy on how to best compress the various part of the data ?
Honestly looks incredible. Could be amazing to provide a general framework for compressing custom format.