Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

151–160 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#151

Post by gcflymoto »

mgerdts wrote: post #17
gpderetta wrote: post #13

The issue with pigz is that uncompressing doesn't really parallelize beyond a three stage read/uncompress/write pipeline. This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.

Would you mind creating a fork of pigz in GitHub and add this patch? I would be interested in testing it out!

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#152

Post by jltsiren »

daneel_w wrote: post #146
jltsiren wrote: post #143

Earlier quoted context omitted.

It was easier back then, because there were fewer people developing new compressors. I've lost count how many compressors have been marketed as a replacement for gzip over the decades. And it's always a replacement for gzip. Every time a new compressor starts getting popular, people start promoting a new even better replacement, and gzip never gets properly replaced. zstd finally has some potential to replace gzip, b…

"zstd finally has some potential to replace gzip" bzip2 and xz have had the potential to replace gzip for the vast majority of users and use cases since more than a decade - and in many cases they have.

What I'm trying to say is that the excessive focus on cutting-edge technology is holding back progress.

gzip is still the default compressor people use when they are not sure about the appropriateness of other compressors in their specific use case, and they don't have the time or energy to find out. To replace it, the a compressor must satisfy two requirements:

* It must not be substantially worse than gzip on any relevant metric. bzip2 failed this by being slow.

* It must be ubiquitous enough that the idea of installing it no longer makes sense. xz never reached this point, before people started replacing it with better compressors.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#154

Post by mikepurvis »

mikepurvis wrote: post #12
osivertsson wrote: post #6

Useful with Docker, see https://github.com/moby/moby/pull/35697 I’ve integrated pigz into different build and CI pipelines a few times. Don’t expect wonders since some steps still need to run serially, but a few seconds here and there might still add up to a few minutes on a large build.

I built a custom dpkg with parallel xz for speeding up the compression of large omni style deb packages. Totally worth it.

Turns out it was longer ago than I thought— way back in the Ubuntu 14.04 and 16.04 days:

https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#155

Post by fis »

ctur wrote: post #53

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.

zstd had a data corruption bug until quite recently. Eventually it may supplant gzip as the de facto standard, but it's too soon to declare it better in every objective way. Give it time.

https://news.ycombinator.com/item?id=35446847

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#156

Post by Taywee »

bick_nyers wrote: post #140
Taywee wrote: post #85

Earlier quoted context omitted.

Economics of scale come into effect as well. Gzip decompression speed is slightly better at higher levels as well. A one time higher cost of compression can pay off pretty quickly when you are decompressing it a lot of times, or serving it to enough people.

I'm not so sure about this. Generally speaking there will be more work done on the CPU to decompress at higher levels (e.g. 6 through 9). It is possible (although unlikely) that you will get higher decompression speed, but only if the bottleneck wasn't CPU to begin with (e.g. network or disc). My gut feeling is that if you are pulling down data faster than 40 Megabits and have a CPU made within the past 7 years (poss…

Most compression algorithms don't take more work to decompress at higher levels, and actually perform better due to having less data to work through. Gzip consistently benchmarks better at decompression for higher levels.

It's not just about bottlenecks, but aggregate energy expenditure from millions of decompressions. On the whole, it can make a real measurable difference. My point was only really that it's not so cut and dry that it's a good trade off to take a 5% file size loss for 20% improved compression performance. You'd have to benchmark and actually estimate the total number of decompressions to see the tipping point.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#157

Post by PaulHoule »

bick_nyers wrote: post #147
PaulHoule wrote: post #142

Earlier quoted context omitted.

In video games long loading times for levels is a serious pain point, so video game developers put a lot of effort into tuning up compression algorithms to get the best wall clock time considering both the time to fetch content from storage and the time to decompress. If the target is a console you may know exactly what hardware is there so you can justify the effort in tuning. (it’s more complex today because you ha…

Don't most games/game engines use TGA format for their textures? Those are all RLE-encoded if I'm not mistaken (which is very fast but very inefficient space-wise). Or perhaps that is just at game creation and those will get baked to some other image format for distribution?

People use all kinds of compression schemes for textures

https://aras-p.info/blog/2020/12/08/Texture-Compression-in-2...

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#158

Post by PaulHoule »

My understanding is that this all works because you can concat two gzip files and the outcome is the same as concatenating the original files

  $ gzip -c a > a.gz
  $ gzip -c b > b.gz
  $ cat a b > c1
  $ cat a.gz b.gz> c.gz
  $ gzip -dc c.gz > c2
  $ cmp c1 c2
  [no output, files match]

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#159

Post by mgerdts »

gcflymoto wrote: post #151
mgerdts wrote: post #17

Earlier quoted context omitted.

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.

Would you mind creating a fork of pigz in GitHub and add this patch? I would be interested in testing it out!

You can grab the version from the solaris userland repo I linked and use it without me completing a homework assignment. Just grab the pigz-2.3.4 source then apply the patches from [1] in the proper order. Maybe some of them aren't needed for non-Solaris.

1. https://github.com/oracle/solaris-userland/tree/master/compo...

I thought I had opened a PR for that a long while ago, but it doesn't show up on github these days. In any case, I did ask Mark Adler to review it. It was never a priority, then the code changed in ways that I don't really want to deal with.

While looking through the PRs, I noticed a PR for Blocked GZip Format (BGZF) [2]. That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in.

2. https://github.com/madler/pigz/pull/19

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#160

Post by FPGAhacker »

ttul wrote: post #77
adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

Side note: In the 1990s, everyone in my engineering school had an HP48 calculator. There were a healthy selection of pretty decent games available. One fine day, I finished my physics exam an hour early and so opened up an enjoyable game on my calculator. 45 minutes went by and so I went up and handed in my paper. It was at this point that the professor noted, “were you planning on leaving the second page blank?” Oh.

Still have mine. For a while on my phone I used an emulator. Eventually I found PCalc and it was customizable enough to recreate the parts of the hp48g that I cared about on a day-to-day basis.
Post reply on HN