41–50 of 70 posts
Why not just use GNU parallel? `parallel gzip ::: file1 file2` I wish there was a standard for this type of stuff. So that an app will check for existing child spawns and act accordingly (IPC).
Earlier quoted context omitted.
With 2 cores (4 logical) 58.9% decrease in time for the 680 MB file a.txt: # time pigz -c /tmp/a.txt > /dev/null real 0m19.352s user 1m16.148s sys 0m0.344s # time gzip -c /tmp/a.txt > /dev/null real 0m47.093s user 0m46.940s sys 0m0.104s
First, thanks for the numbers, it's useful to see real world examples. Second, and this isn't meant to be a critique (I'm just trying to understand phenomena I see), is there a reason you prefer presenting it as a percentage decrease? Every time I read "X% decrease" I feel obliged to read the source numbers because I'm never sure if the person is using the terminology correctly or not (you are), since so often people…
I think teej explained it well. If I say "the new value is +20% or -20%", it is immediately obvious those have the same magnitude and opposite direction. But, for some people, when I say "the new value is 120% or 80% of the old value", it is not immediately obvious that they have the same magnitude. It requires a small extra step for the reader to realize that this means the same amount of relative change.
pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression. With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV. Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations. At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we wi…
My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
The input blocks, while compressed independently, have the last 32K of the previous block loaded as a preset dictionary to preserve the compression effectiveness of deflating in a single thread.
pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
Many decoders simply stop reading after decompressing the first 900k without so much as a peep that something that may astonish you has happened.
[0] http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#hl-interface
[1] https://docs.python.org/2/library/bz2.html#bz2.BZ2FileI learned about pigz in the High Performance MySQL O'Reilly book appendix. I used it, and other techniques there to improve our MySQL backup/restore time by 7x. This in turn won first place at a company hackathon.