Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A parallel implementation of gzip for modern multi-processor multi-core machines

github.com

Post reply on HN
Print view View original

21–30 of 70 posts

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#21

Post by opcenter »

abhishivsaxena wrote: post #17

Any ideas how this would compare to gzip while on a Microserver? I'm thinking of Atom C2750 bare metal from packet.

Looks like that has 8 cores? I would imagine it would make a huge difference depending on other loads on the system. I gave it a try on my file server at home that has an AMD E-350 processor with 2 cores and it shaved off a good 42% of the total time:

  > time gzip -v xubuntu-16.04-desktop-amd64.iso 
  xubuntu-16.04-desktop-amd64.iso:	  1.5% -- replaced with xubuntu-16.04-desktop-amd64.iso.gz
  gzip -v xubuntu-16.04-desktop-amd64.iso  119.07s user 4.66s system 98% cpu 2:05.22 total

  > time pigz -v xubuntu-16.04-desktop-amd64.iso           
  xubuntu-16.04-desktop-amd64.iso to xubuntu-16.04-desktop-amd64.iso.gz 
  pigz -v xubuntu-16.04-desktop-amd64.iso  128.19s user 6.64s system 184% cpu 1:12.97 total

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#23

Post by protomok »

toomuchtodo wrote: post #14
hrehhf wrote: post #9

Earlier quoted context omitted.

I tested it on a 680 MB of text. gzip compresses to 246.0 MB. pigz compresses to 245.5 MB. I see similar percent change on a 3.8 MB text file. So they are approximately equivalent.

What was the difference in speed between runs?

You may be interested in these benchmarks: http://vbtechsupport.com/1614/

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#24

Post by 0x0 »

dorfsmay wrote: post #22

If you know that what you compress is alphabetised text, and space is more important to you than time, then please use lzips over gzips. For a parallel version: http://www.nongnu.org/lzip/plzip.html

Is this gunzip-compatible? If not, then as long as gzip remains the only viable/compatible compression mode for HTTP I think advances in gzip compression is still valuable.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#25

Post by hrehhf »

toomuchtodo wrote: post #14
hrehhf wrote: post #9

Earlier quoted context omitted.

I tested it on a 680 MB of text. gzip compresses to 246.0 MB. pigz compresses to 245.5 MB. I see similar percent change on a 3.8 MB text file. So they are approximately equivalent.

What was the difference in speed between runs?

With 2 cores (4 logical) 58.9% decrease in time for the 680 MB file a.txt:

# time pigz -c /tmp/a.txt > /dev/null

real 0m19.352s user 1m16.148s sys 0m0.344s

# time gzip -c /tmp/a.txt > /dev/null

real 0m47.093s user 0m46.940s sys 0m0.104s

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#26

Post by Rapzid »

Used extensively while a system engineer at a hosting company; 10/10, would use again. Excellent utility if you have the cpu cycles to spare and need to cut time; gzip is almost always your bottleneck. Didn't seem to quite scale linearly, but what does.

--rsyncable support too.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#27

Post by treffer »

Twirrim wrote: post #5

pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/

pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression.

With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV.

Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations.

At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we will see some more bzip2 love in the future due to it's perfect fit for parallelism.

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#29

Post by sambe »

Twirrim wrote: post #5

pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/

There is also pbgzip if you are into indexed access (genome data).

Re: A parallel implementation of gzip for modern multi-processor multi-core machines

#30

Post by kbenson »

hrehhf wrote: post #25
toomuchtodo wrote: post #14

Earlier quoted context omitted.

What was the difference in speed between runs?

With 2 cores (4 logical) 58.9% decrease in time for the 680 MB file a.txt: # time pigz -c /tmp/a.txt > /dev/null real 0m19.352s user 1m16.148s sys 0m0.344s # time gzip -c /tmp/a.txt > /dev/null real 0m47.093s user 0m46.940s sys 0m0.104s

First, thanks for the numbers, it's useful to see real world examples.

Second, and this isn't meant to be a critique (I'm just trying to understand phenomena I see), is there a reason you prefer presenting it as a percentage decrease? Every time I read "X% decrease" I feel obliged to read the source numbers because I'm never sure if the person is using the terminology correctly or not (you are), since so often people mess that up. For myself, I generally use "X ran in Y% of the time Z took." specifically because I don't want people to misinterpret. Is the "X% decrease" presentation preferred/taught, or considered standard? Am I alone in feeling it's more likely to be misinterpreted?

(Sorry your comment is the one I brought this up on, I've just been wondering this for a while.)

Post reply on HN