Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pigz: Parallel gzip for modern multi-processor, multi-core machines

zlib.net

Post reply on HN
Print view View original

101–110 of 197 posts

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#101

Post by mgerdts »

gpderetta wrote: post #69
mgerdts wrote: post #17

Earlier quoted context omitted.

I implemented parallel decompression a while back. It is in Solaris 11.3 and later. https://github.com/oracle/solaris-userland/blob/master/compo... Shortly after submitting a PR the code went through major surgery, and my patch then needed a similar amount of surgery. Oracle then whacked most of the Solaris org, and I don’t think this ever got updated to work with the current pigz.

Nice! You should be able to do it without an index by periodically restarting the dictionary on compression and then looking for something resembling the dictionary, right?

Yeah, probably so at the cost of compatibility. As implemented, the .gz file can be used with `gzip -d`.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#102

Post by zweifuss »

cout wrote: post #83
ElectricalUnion wrote: post #46

Earlier quoted context omitted.

Sometimes you need fast indexed access to a specific file in the compressed content without decompressing the entire file (let's say JARs, that are just ZIPs). TIL: you can use method 93 - Zstandard (zstd) Compression - with ZIPs

93?

See 4.4.5. in https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.8.TX... and here: http://facebook.github.io/zstd/

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#103

Post by PhilipRoman »

dgacmu wrote: post #33
masklinn wrote: post #26

Earlier quoted context omitted.

With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.

You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.

I'm using bzip2 to compress a specific type of backup. In my case I cannot afford to steal CPU time from other running processes, so the backup process runs with severely limited CPU percentage. By crude testing I found that bzip2 used 10x less memory and finished several times faster than xz, while being very close on the compression rate.

Other algorithms like zstd and gz resulted in much lower compression rates.

I'm sure there is a more efficient solution, but changing three letters in a script was pretty much the maximum amount of effort I was going to put in.

On an unrelated note, has someone already made a meta-compression algorithm which simply picks the best performing compression algorithm for each input?

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#104

Post by tehjoker »

elteto wrote: post #60
Aachen wrote: post #51

Earlier quoted context omitted.

I've also noticed for some text documents (was it json? I don't remember) that bzip compresses significantly better than xz (and of course gzip/pigz). Not sure if I tested zstd with high/extreme settings at that time.

For some reason, bzip compresses text incredibly well. And it has for years, I remember noticing this almost 20 years ago.

It uses the Burrows-Wheeler transform to place similar strings next to each other before using other compression tricks, so it usually does a bit better.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#106

Post by kunley »

ctur wrote: post #53

Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.

Not really.

"The recipients" are for example millions of browsers that don't understand zstd.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#107

Post by jandrese »

jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

Shame he didn't discover pbzip2 before starting that job.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#108

Post by res0nat0r »

jimmySixDOF wrote: post #5

John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...

He should just be using pbzip2 :)

https://linux.die.net/man/1/pbzip2

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#109

Post by adamgordonbell »

dralley wrote: post #76
adamgordonbell wrote: post #25

I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…

Fastest open source compression algorithms. RAD game tools have proprietary ones that are faster and have better compression ratios, but since you have to pay for a license, they will never be widespread.

Interesting. Are there any benchmarks you can share? On their website they only compare decompression speed and with zlib and LZMA. It would be interesting to compare to LZ4 HC mode, that unity uses.

Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines

#110

Post by sangnoir »

bick_nyers wrote: post #38
adamgordonbell wrote: post #30

Earlier quoted context omitted.

One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…

People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…

> No one ever tweaks that one setting even though they should

That entirely depends on the use-case. Most people running FFMpeg do it as a once off thing - and if those people like me, when I rip a movie I want the highest quality and lowest size I can get, and I'm happy that the default sacrifices speed for quality and size. The processing can be slow because I'm doing it only once. If you're in the business of encoding video and do it all day everyday, your calculus will be different and you won't be using the defaults regardless.

Post reply on HN