11–20 of 197 posts
Similarly, for bzip2 there is pbzip2 ( http://compression.great-site.net/pbzip2/?i=1 ). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
Useful with Docker, see https://github.com/moby/moby/pull/35697 I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.
This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.
Useful with Docker, see https://github.com/moby/moby/pull/35697 I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.
[1] https://community.ibm.com/community/user/power/blogs/brian-v...
The issue with pigz is that uncompressing doesn't really parallelize beyond a three stage read/uncompress/write pipeline. This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.
https://github.com/oracle/solaris-userland/blob/master/compo...
Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.
Earlier quoted context omitted.
AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P
Similarly, for bzip2 there is pbzip2 ( http://compression.great-site.net/pbzip2/?i=1 ). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
I used pbzip2 on an old octo core xeon server with a decent sas raid and was able to compress at well over 200MB/sec, closer to 300MB in some cases.