OpenZL: An open source format-aware compression framework
1–10 of 110 posts
Re: OpenZL: An open source format-aware compression framework
#2In addition to the blog post, here are the other things we've published today:
Code: https://github.com/facebook/openzl
Documentation: https://openzl.org/
White Paper: https://arxiv.org/abs/2510.03203
Re: OpenZL: An open source format-aware compression framework
#3This is such a leap forward it's hard to believe it's anything but magic.
Re: OpenZL: An open source format-aware compression framework
#4Wow this sounds nuts. I want to try this on some large csvs later today.
Re: OpenZL: An open source format-aware compression framework
#5Wow this sounds nuts. I want to try this on some large csvs later today.
Let us know how it goes!
We developed OpenZL initially for our own consumption at Meta. More recently we've been putting a lot of effort into making this a usable tool for people who, you know, didn't develop OpenZL. Your feedback is welcome!
Re: OpenZL: An open source format-aware compression framework
#6Meta's Nimble is natively integrated with OpenZL (pre-OSS version), and is insanely benefiting from it.
Re: OpenZL: An open source format-aware compression framework
#7Meta's Nimble is natively integrated with OpenZL (pre-OSS version), and is insanely benefiting from it.
Yeah, backend compression in columnar data formats is a natural fit for OpenZL. Knowing the data it is compressing is numeric, e.g. a column of i64 or float, allows for immediate wins over Zstandard.
Re: OpenZL: An open source format-aware compression framework
#8This is such a leap forward it's hard to believe it's anything but magic.
I used to see as magic that the old original compression algorithms worked so well with generic text, without worrying about format, file type, structure or other things that could give hints of additional redundancy.
Re: OpenZL: An open source format-aware compression framework
#9It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!).
Re: OpenZL: An open source format-aware compression framework
#10Cool, but what's the Weissman Score?