Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZL: An open source format-aware compression framework

engineering.fb.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: OpenZL: An open source format-aware compression framework

#101

Post by felixhandte »

adrianmonk wrote: post #99

This is great stuff! Any plans to make it so one format can reference another format? Sometimes data of one type occurs within another format, especially with archive files, media container files, and disk images. So, for example, suppose someone adds a JSON format to OpenZL. Then someone else adds a tar format. While parsing a tar file, if it contains foo.json, there could be some way of saying to OpenZL, "The next…

The OpenZL core supports arbitrary composition of graphs. So you can do this now via the compressor construction APIs. We just have to figure out how to make it easy to do.

Re: OpenZL: An open source format-aware compression framework

#102

Post by felixhandte »

squirrellous wrote: post #82

One of the mentioned examples sounds like the compressor is taking advantage of the SDDL by treating row-oriented data as stripes of column-oriented data, and then compressing that. This makes me curious - for data that’s already column-oriented like Parquet, what’s the advantage of OpenZL over zstd?

SDDL (and the front-end task of reshaping data in general) is only one component of OpenZL. Once you have the streams, you can do all sorts of transformations to them that Zstd doesn't.

Re: OpenZL: An open source format-aware compression framework

#103

Post by p1mrx »

altcognito wrote: post #94
p1mrx wrote: post #93

I tried compressing some CD quality PCM audio: wav=54MB, zstd=51MB, zl=42MB, flac=39MB. So OpenZL is significantly better than zstd, but worse than flac.

Is that with training or without?

I think training is mandatory. These are the commands I used:

https://gist.github.com/pmarks-net/64c17aff45e7741f07eeb5dd0...

Re: OpenZL: An open source format-aware compression framework

#104

Post by p1mrx »

terrelln wrote: post #96
p1mrx wrote: post #93

I tried compressing some CD quality PCM audio: wav=54MB, zstd=51MB, zl=42MB, flac=39MB. So OpenZL is significantly better than zstd, but worse than flac.

Out of curiosity, what was the input file format? We actually worked on a demo WAV compressor a while back. We are currently missing codecs to run the types of predictors that FLAC runs. We expect to add this kind of functionality in the future, in a generic way that isn't specific to audio, and can be used across a variety of domains. But, generally we wouldn't expect to generally beat FLAC. But, be able to offer sp…

The input was just CD audio, "One More Time" by Daft Punk.

test.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, stereo 44100 Hz

Re: OpenZL: An open source format-aware compression framework

#105

Post by TheKaibosh »

ionelaipatioaei wrote: post #90
kingstnap wrote: post #4

Wow this sounds nuts. I want to try this on some large csvs later today.

I must be doing something wrong but I couldn't manage to compressed a file using a custom trained profile since I was getting this error: ``` src/openzl/codecs/dispatch_string/encode_dispatch_string_binding.c:74: EI_dispatch_string: splitting 48000001 strings into 14 outputs OpenZL Library Exception: OpenZL error code: 55 OpenZL error string: Input does not respect conditions for this node OpenZL error context: Code:…

This is unexpected... I'm interested in seeing what's happening here. Do you mind creating a Github issue with as much info as you're comfortable sharing? https://github.com/facebook/openzl/issues

Re: OpenZL: An open source format-aware compression framework

#106

Post by lifthrasiir »

bangaladore wrote: post #97
lifthrasiir wrote: post #76

Earlier quoted context omitted.

As someone seriously trying to develop a compressed archive format with WebAssembly, sandboxing is actually easy and that's indeed why WebAssembly was chosen. The real problem is determinism, which WebAssembly does technically support but actual implementations may vary significantly. And even when WebAssembly can be made fully deterministic, function calls made to those WebAssembly modules may still be undeterminist…

I'm confused why determinism is a problem here? You write an algorithm that should produce the same output for a given input. How does WASM make that not deterministic?

Assume that I have 120 MB of data to process. Since this is quite large, implementations may want to process them in chunks (say, 50 MB). Now those implementations would call the WebAssembly module multiple times with different arguments, and input sizes would depend on the chunk size. Even though each call is deterministic, if you vary arguments non-deterministically then you lose any benefit of determinism: any bug in the WebAssembly module will corrupt data.

Re: OpenZL: An open source format-aware compression framework

#107

Post by bangaladore »

lifthrasiir wrote: post #106
bangaladore wrote: post #97

Earlier quoted context omitted.

I'm confused why determinism is a problem here? You write an algorithm that should produce the same output for a given input. How does WASM make that not deterministic?

Assume that I have 120 MB of data to process. Since this is quite large, implementations may want to process them in chunks (say, 50 MB). Now those implementations would call the WebAssembly module multiple times with different arguments, and input sizes would depend on the chunk size. Even though each call is deterministic, if you vary arguments non-deterministically then you lose any benefit of determinism: any bug…

But that is the case in any language and runtime? There is nothing unique about WASM here.

Re: OpenZL: An open source format-aware compression framework

#108

Post by lifthrasiir »

bangaladore wrote: post #107
lifthrasiir wrote: post #106

Earlier quoted context omitted.

Assume that I have 120 MB of data to process. Since this is quite large, implementations may want to process them in chunks (say, 50 MB). Now those implementations would call the WebAssembly module multiple times with different arguments, and input sizes would depend on the chunk size. Even though each call is deterministic, if you vary arguments non-deterministically then you lose any benefit of determinism: any bug…

But that is the case in any language and runtime? There is nothing unique about WASM here.

Yes and that's exactly my point. It is not enough to make the execution deterministic.

Thinking about that, you may have been confused why I said it's reasonable to avoid WebAssembly for that. I meant that a full Turing-complete execution might not be necessary if that makes it easier to ensure the correctness; OpenZL graphs are not even close to a Turing-complete language for example.

Re: OpenZL: An open source format-aware compression framework

#110

Post by yinnovator »

I am trying to compress a file which has size lot larger than 2 GB , but i am getting error Unhandled Exception: Chunking support is required for compressing inputs larger than 2 GiB. Can't we compress big files with OpenZL , can't find about this error in any documentation
Post reply on HN