Pigz: Parallel gzip for modern multi-processor, multi-core machines
1–10 of 197 posts
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#2Similarly, for bzip2 there is pbzip2 (http://compression.great-site.net/pbzip2/?i=1).
zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#3Similarly, for bzip2 there is pbzip2 ( http://compression.great-site.net/pbzip2/?i=1 ). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#4Similarly, for bzip2 there is pbzip2 ( http://compression.great-site.net/pbzip2/?i=1 ). zstd & xz support the "-T" argument for setting thread count. If you pass "-T 0" it will attempt to detect and use a thread per physical core.
AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#5John Carmack just had a tweet today on this problem:
>I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance.
https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#6Useful with Docker, see https://github.com/moby/moby/pull/35697
I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#7John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#8Earlier quoted context omitted.
AFAIK (not 100% sure), multithreading support is different - parallel versions split the file in multiple segments, and compress each independently, which multithreaded functionalities apply to the same stream (no hard splitting). For this reason, there's for example pzstd, in addition to zstd.
tangential question: compressed files look like hashes in terms of if something changes in the beginning, all the other parts are different, right?
Usually but it depends on the compression scheme. There’s usually a “window” of how far back they look, so they can resync after a while, but it’s unlikely and the offsets will likely have changed so you need to handle that.
You can force this property by introducing synchronisation points though, gzip has an `—-rsyncable` which makes that a lot more likely, at a small compression cost.
Edit: apparently zstd has also had —-rsyncable for the last 5 years.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#9Also works with zlib-ng
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#10John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)
> compressing before tarring is probably better
Not if the files are similar. If you're compressing the files separately you'll start with a clean state rather than reusing previous fragments. Compressing a BMP after a TXT may not be beneficial, but compressing 3 tar'ed TXTs is definitely better than doing them separately.