Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

21–30 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#21

Post by terrelln »

maeln wrote: post #20

So, as I understand, you describe the structure of your data in an SDL and then the compressor can plan a strategy on how to best compress the various part of the data ? Honestly looks incredible. Could be amazing to provide a general framework for compressing custom format.

Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in C++ or Python to parse your format. And this code is compression side only, so the decompressor is agnostic to your format.

[0] https://openzl.org/api/c/graphs/sddl/

Re: OpenZL: An open source format-aware compression framework

#22

Post by ttoinou »

modeless wrote: post #16
ttoinou wrote: post #14

Is this similar to Basis ? https://github.com/BinomialLLC/basis_universal

No, not really. They are both cool but solve different problems. The problem Basis solves is that GPUs don't agree on which compressed texture formats to support in hardware. Basis is a single compressed format that can be transcoded to almost any of the formats GPUs support, which is faster and higher quality than e.g. decoding a JPEG and then re-encoding to a GPU format.

Thanks. I thought basis also had specific encoders depending on the typical average / nature of the data input, like this OpenZL project

Re: OpenZL: An open source format-aware compression framework

#23

Post by d33 »

I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.

Re: OpenZL: An open source format-aware compression framework

#24

Post by maeln »

terrelln wrote: post #21
maeln wrote: post #20

So, as I understand, you describe the structure of your data in an SDL and then the compressor can plan a strategy on how to best compress the various part of the data ? Honestly looks incredible. Could be amazing to provide a general framework for compressing custom format.

Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in C++ or Python to parse your format. And this code is compression side only, so the decompressor is agnostic to your format. [0] https://openzl.org/api/c/graphs/sddl/

Now I cannot stop thinking about how I can fit this somewhere in my work hehe. ZStandard already blew me away when it was released, and this is just another crazy work. And being able to access this kind of state-of-the-art algo' for free and open-source is the oh so sweet cherry on top

Re: OpenZL: An open source format-aware compression framework

#25

Post by artemisart »

d33 wrote: post #23

I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.

I may be misunderstanding the question but that should be just decompressing gzip & compressing with something better like zstd (and saving the gzip options to compress it back), however it won't avoid compressing and decompressing gzip.

Re: OpenZL: An open source format-aware compression framework

#27

Post by Gethsemane »

felixhandte wrote: post #9

It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827

I'd love to see some benchmarks for this on some common genomic formats (fa, fq, sam, vcf). Will be doubly interesting to see its applicability to nanopore data - lots of useful data is lost because storing FAST5/POD5 is a pain.

Re: OpenZL: An open source format-aware compression framework

#28

Post by michalsustr »

Are you thinking about adding stream support? I.e something along the lines of i) build up efficient vocabulary up front for the whole data and then ii) compress by chunks, so it can be decompressed by chunks as well. This is important for seeking in data and stream processing.

Re: OpenZL: An open source format-aware compression framework

#29

Post by felixhandte »

michalsustr wrote: post #28

Are you thinking about adding stream support? I.e something along the lines of i) build up efficient vocabulary up front for the whole data and then ii) compress by chunks, so it can be decompressed by chunks as well. This is important for seeking in data and stream processing.

Yes, definitely! Chunking support is currently in development. Streaming and seeking and so on are features we will certainly pursue as we mature towards an eventual v1.0.0.

Re: OpenZL: An open source format-aware compression framework

#30

Post by jayknight »

Gethsemane wrote: post #27
felixhandte wrote: post #9

It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827

I'd love to see some benchmarks for this on some common genomic formats (fa, fq, sam, vcf). Will be doubly interesting to see its applicability to nanopore data - lots of useful data is lost because storing FAST5/POD5 is a pain.

And a comparison between CRAM and openzl on a sam/bam file. Is openzl indexable, where you can just extract and decompress the data you need from a file if you know where it is?
Post reply on HN