I learned about pigz in the High Performance MySQL O'Reilly book appendix. I used it, and other techniques there to improve our MySQL backup/restore time by 7x. This in turn won first place at a company hackathon.
A parallel implementation of gzip for modern multi-processor multi-core machines
11–20 of 70 posts
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#12This post is a test.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#13And the same for LZMA: https://github.com/vasi/pixz (it's relatively easy to remember those commands)
Doesn't xz already support threads out of the box, by the --threads flag?
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#14My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
I tested it on a 680 MB of text. gzip compresses to 246.0 MB. pigz compresses to 245.5 MB. I see similar percent change on a 3.8 MB text file. So they are approximately equivalent.
What was the difference in speed between runs?
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#15And the same for LZMA: https://github.com/vasi/pixz (it's relatively easy to remember those commands)
Doesn't xz already support threads out of the box, by the --threads flag?
The first stable version of xz with threading was 5.2.0, released in December 2014. A lot of people are running older packages, or were until recently, and habitually use pixz instead.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#16And the same for LZMA: https://github.com/vasi/pixz (it's relatively easy to remember those commands)
The main purpose of this "pixz" appears to be its chunking of the compressed data so that is it partially decompressible (i.e. random access). "xz" has -T/--threads= already for multithreaded processing (although it does seem like pixz has a different default value "all cores" instead of "1 thread") .
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#17Any ideas how this would compare to gzip while on a Microserver? I'm thinking of Atom C2750 bare metal from packet.
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#18I'm using pbzip2 to solve this problem. http://compression.ca/pbzip2/
I was doing that too, until I tried lbzip2 ( http://lbzip2.org/). Does basically the same, also a drop-in replacement, but is even faster by a noticeable amount!
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#19Test
Re: A parallel implementation of gzip for modern multi-processor multi-core machines
#20Earlier quoted context omitted.
Doesn't xz already support threads out of the box, by the --threads flag?
The first stable version of xz with threading was 5.2.0, released in December 2014. A lot of people are running older packages, or were until recently, and habitually use pixz instead.
Current versions of Debian and Ubuntu ship 5.1.1 fwiw, so it's still quite a large number of people. In fact 5.2.0 doesn't even seem to be in Debian unstable yet. Not 100% sure why, but browsing through the wishlist bug open for years [1], it seems to be due to an unfortunately common reason: even widely used open-source software sometimes has surprisingly few maintainers, in this case seemingly one person, who was maintaining the xz package for years but got busy with other things, and nobody else has picked it up. The good-ish news is that the 5.1.x version now seems to be growing old enough that it's starting to block other packages' upgrades (because their upstream now assumes a newer version), which will probably cause enough other Debian maintainers to notice for the situation to be sorted out. Though I did of course use passive voice in that last sentence, as I wait for "the situation to be sorted out".
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731634