pigz: A parallel implementation of gzip for multi-core machines • Hacker News Board

Re: pigz: A parallel implementation of gzip for multi-core machines

#31

Post by dspillett » Mon, Oct 17, 2022, 9:25 PM UTC

walrus01 wrote: post #27

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-...

pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later.

It works because gzip streams can be tracked together as a single stream, at the start of each block is an instruction to reset the compression dictionary as if it is the start of a file/stream (which in practise it is) so you just have to concatenate the parts coming out of the parallel threads in the right order. These resets cause a small drop in overall compression rates but this is small and can be minimised by using large enough blocks.

Re: pigz: A parallel implementation of gzip for multi-core machines

#32

Post by XCSme » Mon, Oct 17, 2022, 9:25 PM UTC

rcarmo wrote: post #29

I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.

I also thought the name was clever, but your comment made it even more interesting. Also, my first thought was, "is this safe to use?", I heard of gzip vulnerabilities before, but a parallel implementation sounds a lot easier to get wrong.

Re: pigz: A parallel implementation of gzip for multi-core machines

#33

Post by danuker » Mon, Oct 17, 2022, 9:26 PM UTC

xfalcox wrote: post #30

One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.

Have you optimized the low-hanging fruit in your image size?

Because compression programs are as high-hanging fruit as you can get, and parallelizing them can only be done once.

Re: pigz: A parallel implementation of gzip for multi-core machines

#34

Post by walrus01 » Mon, Oct 17, 2022, 9:27 PM UTC

dspillett wrote: post #31

walrus01 wrote: post #27

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-...

pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block…

yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.

Re: pigz: A parallel implementation of gzip for multi-core machines

#35

Post by sitkack » Mon, Oct 17, 2022, 9:31 PM UTC

ac29 wrote: post #17

sitkack wrote: post #8

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.…

bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)

Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...

Re: pigz: A parallel implementation of gzip for multi-core machines

#36

Post by booi » Mon, Oct 17, 2022, 9:41 PM UTC

walrus01 wrote: post #28

sitkack wrote: post #8

If you really want to enable all cores for compression and decompression, give pbzip2 a try. pigz isn't as parallel as pbzip2 http://compression.ca/pbzip2/ *edit, as ac29 mentions below, just use zstdmt. In my quick testing it is approximately 8x faster than pbzip2 and gives better compression ratios. Wall clock time went from 41s to 3.5s for a 3.6GB tar of source, pdfs and images AND the resulting file was smaller.…

on the other hand, bzip2 is pretty much obsoleted now by xzip

What is xzip? are you talking about xz?

Re: pigz: A parallel implementation of gzip for multi-core machines

#37

Post by walrus01 » Mon, Oct 17, 2022, 9:43 PM UTC

booi wrote: post #36

walrus01 wrote: post #28

Earlier quoted context omitted.

on the other hand, bzip2 is pretty much obsoleted now by xzip

What is xzip? are you talking about xz?

yes, xz

section 3.6 here

https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mar...

https://en.wikipedia.org/wiki/XZ_Utils

Re: pigz: A parallel implementation of gzip for multi-core machines

#38

Post by dima_vm » Mon, Oct 17, 2022, 9:51 PM UTC

omoikane wrote: post #10

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).

Re: pigz: A parallel implementation of gzip for multi-core machines

#39

Post by Genbox » Mon, Oct 17, 2022, 9:59 PM UTC

dima_vm wrote: post #38

omoikane wrote: post #10

The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.

But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).

Kotlin does have exceptions[1]

[1] https://kotlinlang.org/docs/exceptions.html#java-interoperab...

Re: pigz: A parallel implementation of gzip for multi-core machines

#40

Post by dspillett » Mon, Oct 17, 2022, 9:59 PM UTC

XCSme wrote: post #32

rcarmo wrote: post #29

I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.

I also thought the name was clever, but your comment made it even more interesting. Also, my first thought was, "is this safe to use?", I heard of gzip vulnerabilities before, but a parallel implementation sounds a lot easier to get wrong.

Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream.

This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order.

It is the very same property of the format that gzip's own --rsyncable option makes use of to stop small changes forcing a full file send when rsync (or similar) is used to transfer updated files.

The idea is as simple as it is clever, one of those "why did I not think about that?" ideas that are obvious once someone else has thought of it, so adds little or no extra risk. A vulnerability that uses gzip (a "compression bomb") or can cause a gzip tool to errantly run arbitrary code, is no more likely to affect pigz than it is the standard gzip builds.