Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

191–197 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#191

Post by lionkor »

adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

I loved this episode, it was very engaging to the very end, I wish there were more episodes! (I already listened to all of them so far) Thank you for doing this podcast!

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#192

Post by lionkor »

boomboomsubban wrote: post #28
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

Is the .07 of a core a margin of error or some kind of status report done on a different core?

iirc its because core usage is an average over time, and you can get weird situations where switching between cores (which happens frequently depending on the scheduler) causes the average usage of one core by the program to not be zero yet, and the usage on another core to already be 100.

This is purely speculation though

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#194

Post by account42 »

PhilipRoman wrote: post #99
adamgordonbell wrote: post #40

Earlier quoted context omitted.

> What's well explored is compression rate not performance Exactly! And this seems like a shame to me with something burning so many cpu cycles. > true innovations, like arithmetic coding, are rare. Yeah, Yann tried to explain arithmetic coding to me, but I didn't get it.

I think arithmetic coding is much simpler than the way most resources describe it (the hard part is making it efficient). Consider this example: you want to transmit two completely random variables, both of which can have 5 states. The obvious way is to concatenate two bit fields of size ceil(log2(5)), so 3+3 = 6 bits. But alternatively, you can count the total number of states possible for both variables together, 5…

> The coolest part IMO is how easy it is to plug in custom models for symbol probabilities. Usually a simple counter for each symbol is enough, but you can go crazy and start predicting the next symbol based on previous ones.

PPM-based [0] compression can work quite well for text but on its own it's not enough to unseat the Lempel-Ziv family as the top general purpose compressor.

[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#195

Post by account42 »

gruturo wrote: post #75
jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

With all respect to John Carmack (and it is really a lot of respect!) I'm surprised he seems unaware of pbzip2? It's a parallel implementation scales almost linearly with the amount of cores, and has been around since ~2010, so it's not yet old enough to drive, but anyone dealing with bzip2'ing large amounts of data should have discovered it long ago. And yes, use zstandard (or xz, where the default binary in your di…

> anyone dealing with bzip2'ing large amounts of data

... should really reconsider their choice of compression algorithm at this point.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#196

Post by account42 »

ElectricalUnion wrote: post #46
gpderetta wrote: post #7

Earlier quoted context omitted.

But pigz shows that the unix pipeline philosophy works just fine. (of course compressing before tarring is probably better than compressing the tarred file, but that should be pipelinable as well)

Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs). TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs

The only point of using .zip is for maximum compatibility and you loose that if you use anything other than deflate or no compression. If you are going to use something else you might as well use a less wasteful and better defined archive container - or something entirely custom like most games do.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#197

Post by PhilipRoman »

account42 wrote: post #194
PhilipRoman wrote: post #99

Earlier quoted context omitted.

I think arithmetic coding is much simpler than the way most resources describe it (the hard part is making it efficient). Consider this example: you want to transmit two completely random variables, both of which can have 5 states. The obvious way is to concatenate two bit fields of size ceil(log2(5)), so 3+3 = 6 bits. But alternatively, you can count the total number of states possible for both variables together, 5…

> The coolest part IMO is how easy it is to plug in custom models for symbol probabilities. Usually a simple counter for each symbol is enough, but you can go crazy and start predicting the next symbol based on previous ones. PPM-based [0] compression can work quite well for text but on its own it's not enough to unseat the Lempel-Ziv family as the top general purpose compressor. [0] https://en.wikipedia.org/wiki/Pre…

Yeah arithmetic coding on its own is not enough, typically you would have it as the last step, with dictionary and other compressors before it.
Post reply on HN