Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many times faster (de)compression using multiple processors.

iasylum.net

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Many times faster (de)compression using multiple processors.

#11

Post by isocpprar »

igorhvr wrote: post #9
isocpprar wrote: post #7

Is xz less resource intensive then bzip2? My testing (admittedly two years ago or so) showed significant differences, better compression ratio with xz but significantly longer and/or more memory used.

In the tests I did (including compression of an entire filesystem image), xz always compresses better for the same effort (amount of time). One example (using the standard tools, not the multithreaded ones): igorhvr:tmp/ $ ls -la co_2011-08_import.mdb* -rw-r--r-- 1 igorhvr igorhvr 261832704 2011-10-11 07:19 co_2011-08_import.mdb igorhvr:tmp/ $ sudo time nice -n -20 bzip2 -k co_2011-08_import.mdb 42.85user 0.14system…

Interesting I may have been comparing parallel bzip2 with single threaded xz or perhaps it was just the data I was using for a test.

I'll have to look at it again, thanks!

Re: Many times faster (de)compression using multiple processors.

#12

Post by igorhvr »

getsat wrote: post #10
igorhvr wrote: post #9

Earlier quoted context omitted.

In the tests I did (including compression of an entire filesystem image), xz always compresses better for the same effort (amount of time). One example (using the standard tools, not the multithreaded ones): igorhvr:tmp/ $ ls -la co_2011-08_import.mdb* -rw-r--r-- 1 igorhvr igorhvr 261832704 2011-10-11 07:19 co_2011-08_import.mdb igorhvr:tmp/ $ sudo time nice -n -20 bzip2 -k co_2011-08_import.mdb 42.85user 0.14system…

Put two spaces before your lines of terminal output to prevent HN from munging them.

Done - thanks

Re: Many times faster (de)compression using multiple processors.

#13

Post by alecco »

It's good people get interested in the subject. But this is very odd and has some errors. For example xz requires a lot more memory resources than bzip2 (see benchmarks below, Mem column).

http://mattmahoney.net/dc/text.html

http://mattmahoney.net/dc/uiq/

Matt Mahoney mantains the best benchmarks on text and generic compression. Some of the best on the field (like Matt) usually hang out at encode.ru.

Re: Many times faster (de)compression using multiple processors.

#14

Post by alecco »

joshbaptiste wrote: post #6

Had to try this on my quad core laptop, as I never heard of these tools . josh@snoopy:~/Downloads $ grep -m2 -i intel /proc/cpuinfo vendor_id : GenuineIntel model name : Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz josh@snoopy:~/Downloads $ ls -l test -rw-r--r-- 1 josh josh 1073741824 2012-03-07 20:06 test josh@snoopy:~/Downloads $ time gzip test real 0m16.430s user 0m10.210s sys 0m0.490s josh@snoopy:~/Downloads $ time p…

But what's the compression ratio and what's the test data?

Re: Many times faster (de)compression using multiple processors.

#15

Post by kristianp »

joshbaptiste wrote: post #6

Had to try this on my quad core laptop, as I never heard of these tools . josh@snoopy:~/Downloads $ grep -m2 -i intel /proc/cpuinfo vendor_id : GenuineIntel model name : Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz josh@snoopy:~/Downloads $ ls -l test -rw-r--r-- 1 josh josh 1073741824 2012-03-07 20:06 test josh@snoopy:~/Downloads $ time gzip test real 0m16.430s user 0m10.210s sys 0m0.490s josh@snoopy:~/Downloads $ time p…

That's actually a dual core with hyperthreading. (I'm not just being pedantic, I'm curious as to how well hyperthreading works.) http://ark.intel.com/products/52224

I wonder what the speed looks like with two threads? (pigz -p 2) ?

Re: Many times faster (de)compression using multiple processors.

#16

Post by igorhvr »

alecco wrote: post #13

It's good people get interested in the subject. But this is very odd and has some errors. For example xz requires a lot more memory resources than bzip2 (see benchmarks below, Mem column). http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/uiq/ Matt Mahoney mantains the best benchmarks on text and generic compression. Some of the best on the field (like Matt) usually hang out at encode.ru.

By resources I meant wall-time, mostly - what I am optimizing for (I move >200GB zfs snapshots around a lot..). I did not pay attention to memory, specifically - in that light I updated the side note.

What else did you find odd/wrong?

Re: Many times faster (de)compression using multiple processors.

#17

Post by PaulHoule »

If you're handling a lot of data it make sense to hash-partition it on some key and spread it out to a large number of files.

In that case you might have, say, 512 partitions and you can farm out compression, decompression and other tasks to as many CPUs as you want, even other machines in a cluster.

Re: Many times faster (de)compression using multiple processors.

#18

Post by joshbaptiste »

alecco wrote: post #14
joshbaptiste wrote: post #6

Had to try this on my quad core laptop, as I never heard of these tools . josh@snoopy:~/Downloads $ grep -m2 -i intel /proc/cpuinfo vendor_id : GenuineIntel model name : Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz josh@snoopy:~/Downloads $ ls -l test -rw-r--r-- 1 josh josh 1073741824 2012-03-07 20:06 test josh@snoopy:~/Downloads $ time gzip test real 0m16.430s user 0m10.210s sys 0m0.490s josh@snoopy:~/Downloads $ time p…

But what's the compression ratio and what's the test data?

Test data

  josh@snoopy:~/Downloads $ dd if=/dev/zero of=test bs=1024K count=1000
  
  josh@snoopy:~/Downloads $ gzip -l test.gz 
  compressed        uncompressed  ratio   uncompressed_name
  1176024          1048576000  99.9% test.txt
with -p2 it runs ~2 sec slower

Re: Many times faster (de)compression using multiple processors.

#19

Post by alecco »

joshbaptiste wrote: post #18
alecco wrote: post #14

Earlier quoted context omitted.

But what's the compression ratio and what's the test data?

Test data josh@snoopy:~/Downloads $ dd if=/dev/zero of=test bs=1024K count=1000 josh@snoopy:~/Downloads $ gzip -l test.gz compressed uncompressed ratio uncompressed_name 1176024 1048576000 99.9% test.txt with -p2 it runs ~2 sec slower

For that kind of data you can just use RLE instead. It's the trivial case (an edge case in fact).

Re: Many times faster (de)compression using multiple processors.

#20

Post by alecco »

igorhvr wrote: post #16
alecco wrote: post #13

It's good people get interested in the subject. But this is very odd and has some errors. For example xz requires a lot more memory resources than bzip2 (see benchmarks below, Mem column). http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/uiq/ Matt Mahoney mantains the best benchmarks on text and generic compression. Some of the best on the field (like Matt) usually hang out at encode.ru.

By resources I meant wall-time, mostly - what I am optimizing for (I move >200GB zfs snapshots around a lot..). I did not pay attention to memory, specifically - in that light I updated the side note. What else did you find odd/wrong?

You don't mention ratios. Better compression becomes asymptotically/exponentially harder when it gets close to the optimal code size.

Thanks for the shout out. BTW Matt's benchmarks are for all major compression engines, not only bzip2.

Post reply on HN