Because I can can never remember to use pigz I have to have this in my dotfiles:
function ccm() {
tar -cf - $1 | pigz > $1.tar.gz
}Pigz: Parallel gzip for modern multi-processor, multi-core machines
31–40 of 197 posts
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#32Earlier quoted context omitted.
I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.
Unix generally favors processes over threads, at least the old school Unix. Threads are a more recent innovation. The old approach was that programs don't need internal parallelism because you can get it by just piping stuff and relying on the kernel's buffering to keep multiple processes busy. Eg, tar is running on one core dealing with the filesystem, gzip is running on another core compressing stuff. In the early…
Well, if you split the file into chunks you could fan it across cores by compressing each individually.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#33Earlier quoted context omitted.
I wasn't aware the Unix philosophy was to not use multithreading on large jobs that can be parallelized. You can complain about philosophies but this is just using the wrong tool for the job. Complain about bzip if you feel the bzip authors should have made multithreaded implementation for you.
With all due respect to Carmack he’s using bzip in 2023, that’s pretty outdated on every front.
You'd be surprised. There are some workloads - for me, it's geospatial data - where bzip2 clobbers all of the alternatives.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#34Hardware accelerated [1] "GZIP Acceleration with AIX on Power Systems " pigz [1] https://community.ibm.com/community/user/power/blogs/brian-v...
[deleted]
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#35Best of luck with the implementation, but I do hope the authors realise they should avoid naming their software tools like an old-school pornographic film.
I thought we had learned that from the GIMP[1].
[1] https://www.theregister.com/2019/08/28/gimp_open_source_imag...
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#36Earlier quoted context omitted.
How big is that file... I have 2TB files compressed down to ~300GB and gunzip'ing them takes ~2-3 hours. Granted, that's still a long ass time, but not 2-3 days. If anything, I wonder what kind of hard drive John has. If you're reading them off a network drive backed by tape drums it's probably going to take a while ;P
bzip2 is much slower than gzip.
Very bzip2 problem is that even decompression is slower.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#37I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…
> as well explored as data compression
What's well explored is compression rate, where indeed it's difficult to improve, and true innovations, like arithmetic coding, are rare.
Compressing speed on the other hand it's not very interesting to academics, it's more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD.
ZLib and ZStandard are probably in the same complexity class, but with different constant factors, which academics don't care about but which have massive practical consequences.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#38I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…
People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes.
No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems.
If your justification for not serving up larger .png is that not everyone has fast internet, then you should be either detecting and handling that case separately, downscaling the images, and/or serving .jpeg instead.
One time I was using Topaz AI to upscale video, and I spliced that into their ffmpeg filter and took a whole day off of a week long encode. Low hanging fruit.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#39> exploits multiple processors and multiple cores to the hilt when compressing data
As s side note, this isn't always desirable for this class of coders. In some scenarios (like web server) you might want to favor throughput over response time.
Re: Pigz: Parallel gzip for modern multi-processor, multi-core machines
#40Earlier quoted context omitted.
One wild thing is how much performance wins were available compared to ZLib. Pigz is parrellel, but what if you just had a better way to compress and decompress than DEFLATE? When zstd came out – and Brotli before it to a certain extent – they were 3x faster than ZLib with a slightly higher compression ratio. You'd think that such performance jumps in something as well explored as data compression would be hard to co…
> as well explored as data compression What's well explored is compression rate, where indeed it's difficult to improve, and true innovations, like arithmetic coding, are rare. Compressing speed on the other hand it's not very interesting to academics, it's more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD. ZLib and ZStandard are probably…
> What's well explored is compression rate not performance
Exactly! And this seems like a shame to me with something burning so many cpu cycles.
> true innovations, like arithmetic coding, are rare.
Yeah, Yann tried to explain arithmetic coding to me, but I didn't get it.