A parallel implementation of gzip for modern multi-processor multi-core machines
1–10 of 70 posts
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#2My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#3Why doesn't parallelized gzip get more attention? For larger files, it can be quite a pain to see a multicore machine sit mostly idle. Something like this works pretty well, but I've never seen it done outside of my own stuffs:
split -l1000000 --filter='gzip > $FILE.gz' <(zcat large_fille.txt.gz)
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#4How does the strategy for parallelization of gzip in this project differ from the strategy used for LZMA2 parallelism as implemented by 7z?
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#5pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax.
On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#6And the same for LZMA: https://github.com/vasi/pixz
(it's relatively easy to remember those commands)
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#7http://lbzip2.org/ also exists for bzip2. It works really well, especially since bz2 files are split into discreet blocks which can be un/compressed independently.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#8My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
The max window size for zlib is 32 KB, so I don't think the default sharding at 128 KB would change much. You can pass the -b parameter if you find out a bigger shard works better on your data.
If you are looking for details of the design of pigz, there is a very well-documented overview in the source of pigz.c:
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#9My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
I tested it on a 680 MB of text. gzip compresses to 246.0 MB. pigz compresses to 245.5 MB. I see similar percent change on a 3.8 MB text file. So they are approximately equivalent.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#10I'm using pbzip2 to solve this problem.
http://compression.ca/pbzip2/