Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

51–60 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#51

Post by nunobrito »

mappu wrote: post #37
nunobrito wrote: post #19

Well, well. Kind of surprised to see this really good tool that should have been made available a longer time ago since the approach is quite sound. When the data container is understood, the deduplication is far more efficient because now it is targeted. Licensed as BSD-3-Clause, solid C++ implementation, well documented. Will be looking forward to see new developments as more file formats are contributed.

Specialization for file formats is not novel (e.g. 7-Zip uses BCJ2 prefiltering to convert x86 opcodes from absolute to relative JMP instructions), nor is embedding specialized decoder bytecode in the archive (e.g. ZPAQ did this and won a lot of Matt Mahoney's benchmarks) but i think OpenZL's execution here, along with the data description and training system, is really fantastic.

Thanks, I've enjoyed reading more about ZPAQ but their main focus seems to be versioning (which is quite a useful feature too, will try it later) but they don't include specialized compression per context.

Like you mention, the expandability is quite something. In a few years we might see a very capable compressor.

Re: OpenZL: An open source format-aware compression framework

#52

Post by terrelln »

bede wrote: post #50
terrelln wrote: post #41

Earlier quoted context omitted.

Do you happen to have a pointer to a good open source dataset to look at? Naively and knowing little about CRAM, I would expect that OpenZL would beat Zstd handily out of the box, but need additional capabilities to match the performance of CRAM, since genomics hasn't been a focus as of yet. But it would be interesting to see how much we need to add is generic to all compression (but useful for genomics), vs. techniq…

For BAM this could be a good place to start: https://www.htslib.org/benchmarks/CRAM.html Happy to discuss further

Amazing, thank you!

I will take a look as soon as I get a chance. Looking at the BAM format, it looks like the tokenization portion will be easy. Which means I can focus on the compression side, which is more interesting.

Re: OpenZL: An open source format-aware compression framework

#55

Post by felixhandte »

viraptor wrote: post #53

I wonder, given the docs, how well could AI translate imhex and Kaitai descriptions into SDDL. We could get a few good schemas quickly that way.

Ooh, thanks for mentioning these! I wasn't aware of the existence of these tools but yes it seems very possible that you could transform these other spec formats into SDDL descriptions. I'll check them out.

Re: OpenZL: An open source format-aware compression framework

#57

Post by o11c »

d33 wrote: post #23

I've recently been wondering: could you re-compress gzip to a better compression format, while keeping all instructions that would let you recover a byte-exact copy of the original file? I often work with huge gzip files and they're a pain to work with, because decompression is slow even with zlib-ng.

That's called `pristine-gz`, part of the `pristine-tar` project.

Re: OpenZL: An open source format-aware compression framework

#58

Post by kstenerud »

terrelln wrote: post #36
dist-epoch wrote: post #26

Is this useful for highly repetitive JSON data? Something like stock prices for example, one JSON per line. Unclear if this has enough "structure" for OpenZL.

You'd have to tell OpenZL what your format looks like by writing a tokenizer for it, and annotating which parts are which. We aim to make this easier with SDDL [0], but today is not powerful enough to parse JSON. However, you can do that in C++ or Python. Additionally, it works well on numeric data in native format. But JSON stores it in ASCII. We can transform ASCII integers into int64 data losslessly, but it is ver…

I've done a binary representation of JSON-structured data that uses unary coding for variable length length fields: https://github.com/kstenerud/bonjson/blob/main/bonjson.md#le...

This tends to confuse generic compressors, even though the sub-byte data itself usually clusters around the smaller lengths for most data and thus can be quite repetitive (plus it's super efficient to encode/decode). Could this be described such that OpenZL can capitalize on it?

Re: OpenZL: An open source format-aware compression framework

#60

Post by magicalhippo »

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms.

This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though I assume it's a lighter dependency.

[1]: https://news.ycombinator.com/item?id=45437759 F3: Open-source data file format for the future [pdf] (125 comments)

Post reply on HN