Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#91

Post by blank_state »

TiredOfLife wrote: post #75
magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

And no mention of zpaq that has had emedable decompressors feature for 15 years

you did not read the white paper then

Re: OpenZL: An open source format-aware compression framework

#92

Post by snapplebobapple »

snapplebobapple wrote: post #62
magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

I think this is the first time a genuine technical question of mine rather than a social view has been downvoted here. Thats sad.

Re: OpenZL: An open source format-aware compression framework

#95

Post by touisteur »

terrelln wrote: post #21
maeln wrote: post #20

So, as I understand, you describe the structure of your data in an SDL and then the compressor can plan a strategy on how to best compress the various part of the data ? Honestly looks incredible. Could be amazing to provide a general framework for compressing custom format.

Exactly! SDDL [0] provides a toolkit to do this all with no-code, but today is pretty limited. We will be expanding its feature set, but in the meantime you can also write code in C++ or Python to parse your format. And this code is compression side only, so the decompressor is agnostic to your format. [0] https://openzl.org/api/c/graphs/sddl/

How happy I am to have all written/read data going through a DSL. On to generating the code to make OpenZL happy...

Re: OpenZL: An open source format-aware compression framework

#96

Post by terrelln »

p1mrx wrote: post #93

I tried compressing some CD quality PCM audio: wav=54MB, zstd=51MB, zl=42MB, flac=39MB. So OpenZL is significantly better than zstd, but worse than flac.

Out of curiosity, what was the input file format?

We actually worked on a demo WAV compressor a while back. We are currently missing codecs to run the types of predictors that FLAC runs. We expect to add this kind of functionality in the future, in a generic way that isn't specific to audio, and can be used across a variety of domains.

But, generally we wouldn't expect to generally beat FLAC. But, be able to offer specialized compressors for many types of data that previously weren't important enough to spawn a whole field of specialized compressors, by significantly lowering the bar for entry.

Re: OpenZL: An open source format-aware compression framework

#97

Post by bangaladore »

lifthrasiir wrote: post #76
magicalhippo wrote: post #60

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms. This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though…

As someone seriously trying to develop a compressed archive format with WebAssembly, sandboxing is actually easy and that's indeed why WebAssembly was chosen. The real problem is determinism, which WebAssembly does technically support but actual implementations may vary significantly. And even when WebAssembly can be made fully deterministic, function calls made to those WebAssembly modules may still be undeterminist…

I'm confused why determinism is a problem here? You write an algorithm that should produce the same output for a given input. How does WASM make that not deterministic?

Re: OpenZL: An open source format-aware compression framework

#98

Post by felixhandte »

felixhandte wrote: post #9

It was really hard to resist spilling the beans about OpenZL on this recent HN post about compressing genomic sequence data [0]. It's a great example of the really simple transformations you can perform on data that can unlock significant compression improvements. OpenZL can perform that transformation internally (quite easily with SDDL!). [0] https://news.ycombinator.com/item?id=45223827

Update: let's continue discussing genomic sequence compression on https://github.com/facebook/openzl/issues/76.

Re: OpenZL: An open source format-aware compression framework

#99

Post by adrianmonk »

This is great stuff!

Any plans to make it so one format can reference another format? Sometimes data of one type occurs within another format, especially with archive files, media container files, and disk images.

So, for example, suppose someone adds a JSON format to OpenZL. Then someone else adds a tar format. While parsing a tar file, if it contains foo.json, there could be some way of saying to OpenZL, "The next 1234 bytes are in the JSON format." (Maybe OpenZL's frames would allow making context shifts like this?)

A related thing that would also be nice is non-contiguous data. Some formats include another format but break up the inner data into blocks. For example, a network capture of a TCP stream would include TCP/IP headers, but the payloads of all the packets together constitute another stream of data in a certain format. (This might get memory intensive, though, since there's multiplexing, so you may need to maintain many streams/contexts.)

Re: OpenZL: An open source format-aware compression framework

#100

Post by felixhandte »

bede wrote: post #50
terrelln wrote: post #41

Earlier quoted context omitted.

Do you happen to have a pointer to a good open source dataset to look at? Naively and knowing little about CRAM, I would expect that OpenZL would beat Zstd handily out of the box, but need additional capabilities to match the performance of CRAM, since genomics hasn't been a focus as of yet. But it would be interesting to see how much we need to add is generic to all compression (but useful for genomics), vs. techniq…

For BAM this could be a good place to start: https://www.htslib.org/benchmarks/CRAM.html Happy to discuss further

Wanna hop over to https://github.com/facebook/openzl/issues/76?
Post reply on HN