51–60 of 70 posts
pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
My guess is that this compresses less efficiently as you would have to shard the dictionaries. Might be close though for large files. I was surprised that there were no speed or efficiency comparisons in the README.
The max window size for zlib is 32 KB, so I don't think the default sharding at 128 KB would change much. You can pass the -b parameter if you find out a bigger shard works better on your data. If you are looking for details of the design of pigz, there is a very well-documented overview in the source of pigz.c: https://github.com/madler/pigz/blob/master/pigz.c#L187
Probably useful for servers more than anything where perf really does matter at scale and content type is gzip.
Web servers have multiple connections so this isn't so applicable. Maybe special purpose ones zipping large datasets on the fly?
pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax. On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/
There is also pixz for Xzip format. Which can be a godsend since xzip is such a slow compression. Just make sure you have the CPU to spare since it's a hog.
Earlier quoted context omitted.
pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression. With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV. Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations. At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we wi…
Errors sound deeply concerning. What kind of errors? Is it too risky to use [pl]bzip2 for archival reasons?
But those have been a few years ago, so my best advice would be: retest.
You are testing your archives anyway, right? ;-)
Earlier quoted context omitted.
The max window size for zlib is 32 KB, so I don't think the default sharding at 128 KB would change much. You can pass the -b parameter if you find out a bigger shard works better on your data. If you are looking for details of the design of pigz, there is a very well-documented overview in the source of pigz.c: https://github.com/madler/pigz/blob/master/pigz.c#L187
had a quick scroll thru that source code - i didnt know you could implement a try/catch in C via macros...mind blown.
https://www.amazon.com/Interfaces-Implementations-Techniques...
I love it! I just wish somebody can make a GPU based compression library (doesn't have to be gzip or bzip), every mobile device today is shipping with a GPU, there are some techniques out there like this ( http://on-demand.gputechconf.com/gtc/2014/presentations/S445... ), but I am still waiting for a solid implementation.
Earlier quoted context omitted.
Errors sound deeply concerning. What kind of errors? Is it too risky to use [pl]bzip2 for archival reasons?
Well, errors at a rate of one every few terabyte, and most of the time it was mainly decompression, and always caused program abort iirc. But those have been a few years ago, so my best advice would be: retest. You are testing your archives anyway, right? ;-)
Umm no, I handle errors which I expect to be reported to me.