Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

121–130 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#121

Post by daneel_w »

capableweb wrote: post #58
ctur wrote: post #53

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.

> There literally are no trade-offs; it is better in every objective way. There literally are trade-offs, you started your comment describing one of them. If you want as wide out-of-the-box support as possible, you'd go with gzip. The Compression Streams browser API only supports gzip (+ deflate) so if you wanna compress something natively in the browser without 3rd party libraries (or slow JS implementation), gzip s…

People had your exact sentiments, concerns and hesitations after gzip showed up in the early 90s. Eventually they moved on from pkzip/lzh/etc. to better, modern software - some on their own owed to being reasonable people, and some being dragged along with claws in ground while screaming about "breaking support".

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#123

Post by daneel_w »

bityard wrote: post #66
ctur wrote: post #53

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.

gzip has the advantage of being ubiquitous. It's pretty much guaranteed to be available on every modern Unix-alike. And is good enough for most purposes. Zstd is getting there but I personally don't bother with it on a daily basis except in situations where both performance and compression ratio are important, like build artifact pipelines or large archives.

You're reading old comp-sci usenet discussions from 1993 and you come across this statement: "pkzip/lzh/arc/zoo have the advantage of being ubiquitous. We should not encourage the use of gzip". You chortle.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#124

Post by jandrese »

zargon wrote: post #118
jandrese wrote: post #107

Earlier quoted context omitted.

Shame he didn't discover pbzip2 before starting that job.

If it's less than 98% complete he could still stop it, start over and still finish sooner.

I'd bet tar becomes the bottleneck before pbzip2 does on that multicore monster. It can be surprisingly slow and I don't think any version of tar uses more than one core.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#125

Post by Retr0id »

mgerdts wrote: post #52

If you are interested in optimizing parallel decompression and you happen to have a suitable NVIDIA GPU, GDeflate [1] is interesting. The target market for this is PC games using DirectStorage to quickly load game assets. The graph in [1] shows DirectStorage maxing out the throughput of a PCIe Gen 3 drive at about 3 GiB/s when compression is not used. When GPU GDeflate is used, the effective rate hits 12 GiB/s. If yo…

> the effective rate hits 12 GiB/s

I assume this is for decompressing multiple independent deflate streams in parallel?

What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#127

Post by lkuty »

kps wrote: post #44
fleetside72 wrote: post #23

I was mostly interested in the name and the pronunciation section kind of ruined it for me

If it's to be pig-zee (pixie) for the Americans, it can be pig-zed (pig's head) for the rest of us.

That's funny. Exactly, in french we pronounce z like zed and so that allows us to keep the funny part. Even funnier than "pigs". Thanks

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#128

Post by pdw »

adamgordonbell wrote: post #30
adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

zlib/gzip did not choose DEFLATE because it was the best algorithm. Rather it was, at the time, the only decent algorithm that could be implemented in a manner not covered by patents. (See the tragedy of LZW for why that was important.)

We're now more than two decades later, so all the important data compression patents should have expired.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#129

Post by mgerdts »

Retr0id wrote: post #125
mgerdts wrote: post #52

If you are interested in optimizing parallel decompression and you happen to have a suitable NVIDIA GPU, GDeflate [1] is interesting. The target market for this is PC games using DirectStorage to quickly load game assets. The graph in [1] shows DirectStorage maxing out the throughput of a PCIe Gen 3 drive at about 3 GiB/s when compression is not used. When GPU GDeflate is used, the effective rate hits 12 GiB/s. If yo…

> the effective rate hits 12 GiB/s I assume this is for decompressing multiple independent deflate streams in parallel? What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)

The data is compressed with GDeflate, not deflate. The single stream is designed to use the parallelism of a GPU. It is described here:

https://github.com/microsoft/DirectStorage/blob/main/GDeflat...

The GPU decompression benchmark I linked earlier allows you to specify a single file that it will compress with GDeflate (and zlib for comparison). The numbers presented in the docs that come with the benchmark and presented elsewhere are consistent with my own runs using a source file that is highly compressible.

Part of the trick of achieving this speedup is to read the data fast enough. I don't know of any NVMe drive that can reach full speed with a queue depth of 1. While running the benchmark in a windows VM with a GPU passed through, on the linux host I observed that the average read size was about 512k and the queue depth was sometimes over 30.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#130

Post by mgerdts »

Retr0id wrote: post #125
mgerdts wrote: post #52

If you are interested in optimizing parallel decompression and you happen to have a suitable NVIDIA GPU, GDeflate [1] is interesting. The target market for this is PC games using DirectStorage to quickly load game assets. The graph in [1] shows DirectStorage maxing out the throughput of a PCIe Gen 3 drive at about 3 GiB/s when compression is not used. When GPU GDeflate is used, the effective rate hits 12 GiB/s. If yo…

> the effective rate hits 12 GiB/s I assume this is for decompressing multiple independent deflate streams in parallel? What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)

> I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy

You saw this, right?

https://news.ycombinator.com/item?id=35915285

Post reply on HN