11–20 of 107 posts
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
I recall learning the technique from Tumblr's eng blog.
https://engineering.tumblr.com/post/7658008285/efficiently-c...
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
SQL backups were simply a bash script using Pigz, running on a cron job. Simple times!
I used this recently with -0 (no compression) to pack* billions of files into a tar file before sending them over the network. It worked amazing.
Maybe I’m missing something, but why send the tar generated stream through a non-compressing compressor when you could just send the tar directly?
tar --use-compress-program="pigz -0" ...
If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.…
For everything else, there's zstd (also natively multithread)
Earlier quoted context omitted.
Why use tar | pigz -0 when you can just use tar?
I used tar --use-compress-program="pigz" to create the tar out of billions of files