pigz: A parallel implementation of gzip for multi-core machines
1–10 of 107 posts
Re: pigz: A parallel implementation of gzip for multi-core machines
#2Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?
Re: pigz: A parallel implementation of gzip for multi-core machines
#3We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
Re: pigz: A parallel implementation of gzip for multi-core machines
#4Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?
I used it and it was noticeably faster. I didn't write down by how much.
Re: pigz: A parallel implementation of gzip for multi-core machines
#5I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
Re: pigz: A parallel implementation of gzip for multi-core machines
#6Use this all the time (or did when I was doing more sysadminy stuff). Useful in all sorts of backup pipelines
Re: pigz: A parallel implementation of gzip for multi-core machines
#7Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?
Single-threaded gzip can outperform pigz, or at least come very close, when used with GNU xargs on separate files with no dependencies.
https://www.linuxjournal.com/content/parallel-shells-xargs-u...
Re: pigz: A parallel implementation of gzip for multi-core machines
#8If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2
*edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.
megs
3781 test.tar
3041 test.tar.zstd (default compression 3, 3.5s)
3170 test.tar.bz2 (default compression, 8 threads, 40s)Re: pigz: A parallel implementation of gzip for multi-core machines
#9Any comparative benchmarks or a write-up on the approach (other than "uses zlib and pthreads" from the README)?
Single-threaded gzip can outperform pigz, or at least come very close, when used with GNU xargs on separate files with no dependencies. https://www.linuxjournal.com/content/parallel-shells-xargs-u... https://news.ycombinator.com/item?id=26178257
pigz is most useful on a single stream of data, vs. the more obviously parallel case of files without dependencies.
Re: pigz: A parallel implementation of gzip for multi-core machines
#10The bit I found most interesting was actually:
https://github.com/madler/pigz/blob/master/try.h
https://github.com/madler/pigz/blob/master/try.c
which implements try/catch for C99.