Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#31

Post by terrelln »

jayknight wrote: post #30
Gethsemane wrote: post #27

Earlier quoted context omitted.

I'd love to see some benchmarks for this on some common genomic formats (fa, fq, sam, vcf). Will be doubly interesting to see its applicability to nanopore data - lots of useful data is lost because storing FAST5/POD5 is a pain.

And a comparison between CRAM and openzl on a sam/bam file. Is openzl indexable, where you can just extract and decompress the data you need from a file if you know where it is?

> Is openzl indexable

Not today. However, we are considering this as we are continuing to evolve the frame format, and it is likely we will add this feature in the future.

Re: OpenZL: An open source format-aware compression framework

#33

Post by mappu »

d33 wrote: post #23

I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.

precomp/antix/... are tools that can bruteforce the original gzip parameters and let you recreate the byte-identical gzip archive.

The output is something like {precomp header}{gzip parameters}{original uncompressed data} which you can then feed to a stronger compressor.

A major use case is if you have a lot of individually gzipped archives with similar internal content, you can precomp them and then use long-range solid compression over all your archives together for massive space savings.

Re: OpenZL: An open source format-aware compression framework

#34

Post by modeless »

ttoinou wrote: post #22
modeless wrote: post #16

Earlier quoted context omitted.

No, not really. They are both cool but solve different problems. The problem Basis solves is that GPUs don't agree on which compressed texture formats to support in hardware. Basis is a single compressed format that can be transcoded to almost any of the formats GPUs support, which is faster and higher quality than e.g. decoding a JPEG and then re-encoding to a GPU format.

Thanks. I thought basis also had specific encoders depending on the typical average / nature of the data input, like this OpenZL project

It probably does have different modes that it selects based on the input data. I don't know that much about the implementation of image compression, but I know that PNG for example has several preprocessing modes that can be selected based on the image contents, which transform the data before entropy encoding for better results.

The difference with OpenZL IIUC seems to be that it has some language that can flexibly describe a family of transformations, which can be serialized and included with the compressed data for the decoder to use. So instead of choosing between a fixed set of transformations built into the decoder ahead of time, as in PNG, you can apply arbitrary transformations (as long as they can be represented in their format).

Re: OpenZL: An open source format-aware compression framework

#36

Post by terrelln »

dist-epoch wrote: post #26

Is this useful for highly repetitive JSON data? Something like stock prices for example, one JSON per line. Unclear if this has enough "structure" for OpenZL.

You'd have to tell OpenZL what your format looks like by writing a tokenizer for it, and annotating which parts are which. We aim to make this easier with SDDL [0], but today is not powerful enough to parse JSON. However, you can do that in C++ or Python.

Additionally, it works well on numeric data in native format. But JSON stores it in ASCII. We can transform ASCII integers into int64 data losslessly, but it is very hard to transform ASCII floats into doubles losslessly and reliably.

However, given the work to parse the data (and/or massage it to a more friendly format), I would expect that OpenZL would work very well. Highly repetitive, numeric data with a lot of structure is where OpenZL excels.

[0] https://openzl.org/api/c/graphs/sddl/

Re: OpenZL: An open source format-aware compression framework

#37

Post by mappu »

nunobrito wrote: post #19

Well, well. Kind of surprised to see this really good tool that should have been made available a longer time ago since the approach is quite sound. When the data container is understood, the deduplication is far more efficient because now it is targeted. Licensed as BSD-3-Clause, solid C++ implementation, well documented. Will be looking forward to see new developments as more file formats are contributed.

Specialization for file formats is not novel (e.g. 7-Zip uses BCJ2 prefiltering to convert x86 opcodes from absolute to relative JMP instructions), nor is embedding specialized decoder bytecode in the archive (e.g. ZPAQ did this and won a lot of Matt Mahoney's benchmarks) but i think OpenZL's execution here, along with the data description and training system, is really fantastic.

Re: OpenZL: An open source format-aware compression framework

#38

Post by jltsiren »

Gethsemane wrote: post #27
felixhandte wrote: post #9

It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827

I'd love to see some benchmarks for this on some common genomic formats (fa, fq, sam, vcf). Will be doubly interesting to see its applicability to nanopore data - lots of useful data is lost because storing FAST5/POD5 is a pain.

OpenZL compressed SAM/BAM vs. CRAM is the interesting comparison. It would really test the flexibility of the framework. Can OpenZL reach the same level of compression, and how much effort does it take?

I would not expect much improvement in compressing nanopore data. If you have a useful model of the data, creating a custom compressor is not that difficult. It takes some effort, but those formats are popular enough that compressors using the known models should already exist.

Post reply on HN