Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A parallel implementation of gzip for modern multi-processor multi-core machines

github.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#32

Post by vasi »

0x0 wrote: post #16
tobias3 wrote: post #6

And the same for LZMA: https://github.com/vasi/pixz (it's relatively easy to remember those commands)

The main purpose of this "pixz" appears to be its chunking of the compressed data so that is it partially decompressible (i.e. random access). "xz" has -T/--threads= already for multithreaded processing (although it does seem like pixz has a different default value "all cores" instead of "1 thread") .

xz in multithreaded mode supports random access too, at least theoretically. But there's no reasonable way with xz to actually find the file in a tarball you want to access, it's that bit that pixz provides.

Another nice thing about pixz is it does parallel decompression, as well as compression.

(Disclaimer: I'm the original author of pixz.)

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#33

Post by eyelidlessness »

0x0 wrote: post #24
dorfsmay wrote: post #22

If you know that what you compress is alphabetised text, and space is more important to you than time, then please use lzips over gzips. For a parallel version: http://www.nongnu.org/lzip/plzip.html

Is this gunzip-compatible? If not, then as long as gzip remains the only viable/compatible compression mode for HTTP I think advances in gzip compression is still valuable.

I don't think the comment you replied to suggested that gzip is not valuable, only that there's specialized compression available for a special case (alphabetized data).

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#34

Post by MichaelGG »

Hydraulix989 wrote: post #31

Probably useful for servers more than anything where perf really does matter at scale and content type is gzip.

Web servers have multiple connections so this isn't so applicable. Maybe special purpose ones zipping large datasets on the fly?

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#36

Post by pselbert »

It's amazing to see what somebody like Mark Adler can achieve when they focus on a specific niche. You often hear about the importance of specialization when positioning a business, but it isn't mentioned as much for individuals. His work serves as an ideal case of specialization—if I wanted to ask an expert for advice on compression, or was looking for an outside expert on compression that is immediately who I'd think of.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#37

Post by mgerdts »

pigz parallelizes compression. I've made changes that make it so that it can parallelize uncompression as well.

https://github.com/mgerdts/pigz

This feature is present in Solaris 11.3 and later. Actually, it is in some later patches to 11.2 as well. I added it to speed up suspend and resume of kernel zones.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#38

Post by teej »

kbenson wrote: post #30
hrehhf wrote: post #25

Earlier quoted context omitted.

With 2 cores (4 logical) 58.9% decrease in time for the 680 MB file a.txt: # time pigz -c /tmp/a.txt > /dev/null real 0m19.352s user 1m16.148s sys 0m0.344s # time gzip -c /tmp/a.txt > /dev/null real 0m47.093s user 0m46.940s sys 0m0.104s

First, thanks for the numbers, it's useful to see real world examples. Second, and this isn't meant to be a critique (I'm just trying to understand phenomena I see), is there a reason you prefer presenting it as a percentage decrease? Every time I read "X% decrease" I feel obliged to read the source numbers because I'm never sure if the person is using the terminology correctly or not (you are), since so often people…

I always state as relative change. I find that people can get really confused if you were to say "X ran in 110% in the time of Y" even though it is stated in a clear way.

My preferred way of communicating this concept is "we observed a +10% change in X compared to Y." I always use a +/- sign and this helps signal that I'm talking about a relative change.

If I am comparing percents, I'll always specify "relative" or "absolute" change though I prefer to use relative change. Occasionally if the change is small, I will use basis points instead of percents to communicate absolute change.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#39

Post by mgerdts »

0x0 wrote: post #16
tobias3 wrote: post #6

And the same for LZMA: https://github.com/vasi/pixz (it's relatively easy to remember those commands)

The main purpose of this "pixz" appears to be its chunking of the compressed data so that is it partially decompressible (i.e. random access). "xz" has -T/--threads= already for multithreaded processing (although it does seem like pixz has a different default value "all cores" instead of "1 thread") .

The --index option that I added (see another comment) to pigz allows random access so that uncompression could be multi-threaded.

The workload I was interested in (compressing virtual machine memory during suspend) tends to be I/O bound. pigz struck a good balance to make it so that cpus and disks both stayed busy in an important subset of the machines of interest. This balanced saturation seemed to be optimal to minimize the suspend time. If the suspend image was on fast storage (that is, could saturate a 10 gig link) multi-threading uncompression for resume made a big difference.

I tried pixz and I think pbzip2 and found that while the compress ratio was better than pigz offers, the delta in CPU-bound run time during compression was unacceptable.

The scales may tip in favor of pixz when compression speed is less important than (e.g.) minimizing the amount of bandwidth required for 1000's of downloads of the compressed content.

Post reply on HN