Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

21–30 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#21

Post by Aachen »

jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized.

You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#22

Post by lathiat »

aigoochamna wrote: post #19
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

How big is that file... I have 2TB files compressed down to ~300GB and gunzip'ing them takes ~2-3 hours. Granted, that's still a long ass time, but not 2-3 days. If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P

bzip2 is much slower than gzip.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#24

Post by cout »

gpderetta wrote: post #7
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

For many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio.

Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compression settings with solid archive turned off). Tarring then compressing means small files can be aggregated into a single block, giving both good speed and compression ratio.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#25

Post by adamgordonbell »

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd.

If you'll excuse the plug, here is the LZ4 story:

Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator.

Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders.

His code ended up everywhere: in games, databases, file systems, and the Linux Kernel because Yann built the world's fastest compression algorithms. And he got started just making a fun game for a graphing calculator he'd had since high school.

https://corecursive.com/data-compression-yann-collet/

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#26

Post by masklinn »

Aachen wrote: post #21
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.

With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#27

Post by dale_glass »

Aachen wrote: post #21
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.

Unix generally favors processes over threads, at least the old school Unix. Threads are a more recent innovation.

The old approach was that programs don't need internal parallelism because you can get it by just piping stuff and relying on the kernel's buffering to keep multiple processes busy.

Eg, tar is running on one core dealing with the filesystem, gzip is running on another core compressing stuff.

In the early days, Windows would have a single program doing everything (eg, Winzip) on a single core, while Unix would have a pipeline with the task split into multiple programs that would allow for this implicit parallelism and perform noticeably better.

Today this is all old and doesn't cut it anymore on 128 core setups.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#28

Post by boomboomsubban »

jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

Is the .07 of a core a margin of error or some kind of status report done on a different core?

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#29

Post by zweifuss »

viraptor wrote: post #10
gpderetta wrote: post #7

Earlier quoted context omitted.

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

> compressing before tarring is probably better Not if the files are similar. If you're compressing the files separately you'll start with a clean state rather than reusing previous fragments. Compressing a BMP after a TXT may not be beneficial, but compressing 3 tar'ed TXTs is definitely better than doing them separately.

That is true and seems easily half the total size with small and similar files, but it also means you have to unpack the whole archive when you need the last file in a tarball.

AFAIK the gzip command still cannot compress directory information and therefore needs tar in front of it if you want to retain a folder structure.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#30

Post by adamgordonbell »

adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE?

When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to come by. We weren't that close to the efficiency frontier.

Post reply on HN