Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

41–50 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#41

Post by cpuguy83 »

macNchz wrote: post #14
osivertsson wrote: post #6

Useful with Docker, see https://github.com/moby/moby/pull/35697 I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.

Am I reading correctly that Docker just automatically uses pigz if it’s in the system path? I’ve used both for years and had no idea. I’m definitely going to make sure it’s installed in CI pipelines going forward, I know of some bloated image builds it will definitely help with!

Correct, if it detects in unpigz it will use it. It will not compress layers with it, though.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#42

Post by jdblair »

I'm a big fan of pigz. I use it in my home-grown backup script for my Linux laptop. It can compress the incremental tar output from my filesystem snapshot fast enough to saturate the I/O to my external USB3 hard drive. This is a low bar, but single-threaded gzip (or bzip2) could not do it!

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#43

Post by sllabres »

jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

There is a _parallel_ bzip2:

http://compression.great-site.net/pbzip2/

which should solve the 'my cores are idle' issue.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#45

Post by ljosifov »

I update/upgrade/switch over to zstd (from older compressors) wherever I'm updating or revamping any of my data pipelines. Looks like a win^3 for me: 1) It's probably either in the top-X or #1 in any of the usual compression metrics size/speed/convenience/ease etc. 2) Can do --rsyncable and create rsync friendly files at tiny size cost. 3) In the rare occasion I need there's $ zstd -c file1 >file.zst; zstd -c file2 >>file.zst, then $ zstd -dc file.zst will produce out $ cat file{1,2}

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#46

Post by ElectricalUnion »

gpderetta wrote: post #7
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs).

TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#48

Post by ElectricalUnion »

ComodoHacker wrote: post #39

> exploits multiple processors and multiple cores to the hilt when compressing data As s side note, this isn't always desirable for this class of coders. In some scenarios (like web server) you might want to favor throughput over response time.

`zstd --adapt` is pretty cool as it detects how much output buffer it has and changes compression effort on the fly to try to achieve maximum throughput.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#49

Post by krzyk »

dgacmu wrote: post #33
masklinn wrote: post #26

Earlier quoted context omitted.

With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.

You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.

Oh, that's interesting, I stopped using bzip2 at the time kernel sources started shipping in xz.

Do you know if there are any tests showing which compressor is better (compression wise) for which data?

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#50

Post by gildas »

Similarly, for zipping files in JS, I have coded the possibility to compress zip files on several cores in zip.js [1]. The approach is simpler as it consists of compressing the entries in parallel. It still offers a significant performance gain though when compressing multiple files in a zip file, which is often the nominal case.

[1] https://github.com/gildas-lormeau/zip.js

Post reply on HN