Pigz: Parallel gzip for modern multi-processor, multi-core machines

Post by henrydark » Fri, May 12, 2023, 2:07 PM UTC

adamgordonbell wrote: post #30

adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

Post by draxil » Fri, May 12, 2023, 2:08 PM UTC

Post by loeg » Fri, May 12, 2023, 2:09 PM UTC

joaonmatos wrote: post #35

Best of luck with the implementation, but I do hope the authors realise they should avoid naming their software tools like an old-school pornographic film. I thought we had learned that from the GIMP[1]. [1] https://www.theregister.com/2019/08/28/gimp_open_source_imag...

Post by tda » Fri, May 12, 2023, 2:14 PM UTC

adamgordonbell wrote: post #40

dist-epoch wrote: post #37

Earlier quoted context omitted.

> as well explored as data compression What's well explored is compression rate, where indeed it's difficult to improve, and true innovations, like arithmetic coding, are rare. Compressing speed on the other hand it's not very interesting to academics, it's more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD. ZLib and ZStandard are probably…

> What's well explored is compression rate not performance Exactly! And this seems like a shame to me with something burning so many cpu cycles. > true innovations, like arithmetic coding, are rare. Yeah, Yann tried to explain arithmetic coding to me, but I didn't get it.

Post by gpderetta » Fri, May 12, 2023, 2:16 PM UTC

mgerdts wrote: post #17

gpderetta wrote: post #13

The issue with pigz is that uncompressing doesn't really parallelize beyond a three stage read/uncompress/write pipeline. This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.