21–30 of 107 posts
Earlier quoted context omitted.
Why use tar | pigz -0 when you can just use tar?
I used tar --use-compress-program="pigz" to create the tar out of billions of files
Earlier quoted context omitted.
I used tar --use-compress-program="pigz" to create the tar out of billions of files
But what’s confusing everyone is that tar cf - will create the tar without any external compression program needed.
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.
Pretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds. I recall learning the technique from Tumblr's eng blog. https://engineering.tumblr.com/post/7658008285/efficiently-c...
Separately, at Tumblr I vaguely remember examining some alternative to pigz that was consistently faster at the time (11 years ago) because pigz couldn't parallelize decompression. Can't quite remember the name of the alternative, but it had licensing restrictions which made it less attractive than pigz.
Edit: the old fast alternative I was thinking of is qpress, formerly hosted at http://www.quicklz.com/ but that's no longer online. Googling it now, there are some mirrors and also looks like Percona tools used/bundled it. Not sure if they still do or if they've since switched to zstd.
Earlier quoted context omitted.
Maybe I’m missing something, but why send the tar generated stream through a non-compressing compressor when you could just send the tar directly?
I didn't have the tar, I created it using: tar --use-compress-program="pigz -0" ...
On Linux would it Just Work™ if you aliased pigz to gzip as a drop-in replacement?
FWIW, I configure BackupPC to use pigz instead of gzip without any issues.
zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources.
https://gregoryszorc.com/blog/2017/03/07/better-compression-...
If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.…