Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

81–90 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#81

Post by mmh0000 »

vr46 wrote: post #31

Because I can can never remember to use pigz I have to have this in my dotfiles: function ccm() { tar -cf - $1 | pigz > $1.tar.gz }

Okay, I feel that my first response was too snarky. I'm sorry. In its place, I'll say:

You wouldn't go outside without pants. You shouldn't use a variable without quotes. Put pants on all variables. Also, you shouldn't use () with 'function' it "works" in Bash but it's not standard:

  ccm() {
    tar c "${1}" | pigz > "${1}.tar.gz"
  }
You could further improve it with a loop to accept multiple files:

  ccm() {
    for i in $@; do
      tar c "${i}" | pigz > "${i}.tar.gz"
    done
  }
Now you can run it like: `ccm file1 file2 file3`

See: https://mywiki.wooledge.org/Quotes

See: https://mywiki.wooledge.org/BashGuide/CompoundCommands#Funct...

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#82

Post by 2809 »

Still blows my mind people still use gzip. 20 years ago I was expecting by this point in time for there to be lots of effort put into increasing compression and working towards getting that fast, instead its been a push for speed. It makes sense with how the internet has changed. These days gzip isn't even in the top 100 as far as compression goes, hell even something like RAR or 7zip are far back compared to the best.

Take something like enwik8 (100megs), gzip will get that down to 36megs, with LZMA down to ~24-25. The top of the line stuff will get it down to the ~15meg range. Thats a huge difference.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#83

Post by cout »

ElectricalUnion wrote: post #46
gpderetta wrote: post #7

Earlier quoted context omitted.

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs). TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs

93?

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#84

Post by 5e92cb50239222b »

polski-g wrote: post #57
ctur wrote: post #53

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.

`tar czf` is a lot easier to remember than `tar -I zstd cf`

GNU tar can autodetect the compression algorithm, both for compression and decompression.

  $ tar -caf dst.tar.zst /src
  $ tar -xaf src.tar.zst
(it's fine to omit -a for decompression)

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#85

Post by Taywee »

bick_nyers wrote: post #38
adamgordonbell wrote: post #30

Earlier quoted context omitted.

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…

Economics of scale come into effect as well. Gzip decompression speed is slightly better at higher levels as well. A one time higher cost of compression can pay off pretty quickly when you are decompressing it a lot of times, or serving it to enough people.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#88

Post by usefulcat »

adamgordonbell wrote: post #30
adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

Probably a lot of it is due to hardware changes since zlib was first written. Mainly the importance of cache and branch prediction, which were either less of a big deal or non-existent back then. IOW, zlib probably leaves a lot more on the table now than when it was written.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#89

Post by fnordpiglet »

pmarreck wrote: post #79
danking00 wrote: post #73

Earlier quoted context omitted.

This episode was fascinating. I had heard of LZ4 but not Zstd. It spurred me to make changes to our system at work that are reducing file sizes by as much as 25%. It’s great to have a podcast in which I learn practical stuff!

Probably the most underrated feature of zstd (likely because it's so unusual) is the ability to create a separate compression dictionary. This allows you to develop customized and highly efficient dictionaries that are highly specific to a type of data AND allow you to compress elements of that data without including an entire separate dictionary in every compression output. So for example take logfiles. You can trai…

I do this to make extraordinarily small UDP packets for a low latency system. I record the raw payload then build a dictionary for the data, then share it on both sides. It reduces the packet overhead by removing the dictionary and it does a much better job than other approaches.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#90

Post by tgtweak »

I remember moving a HUGE mysql table (>500GB) with a pipe chain of mysqldump > pigz > scp (compression disabled) > pigz > mysql

If you've ever screwed around with mysqldump -> tar -> scp -> untar -> mysql<, you'll appreciate the speedup on this, in cases where you're setting up a slave and want to have the freshest possible data before kicking off binlog replication - this is the best.

Post reply on HN