If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them.
pigz: A parallel implementation of gzip for multi-core machines
71–80 of 107 posts
Re: pigz: A parallel implementation of gzip for multi-core machines
#72Earlier quoted context omitted.
But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
Kotlin does have exceptions[1] [1] https://kotlinlang.org/docs/exceptions.html#java-interoperab...
They said "try to get rid of", which to me is akin to de-emphasizing.
Of course you'd have to deal with exceptions since you're running on the JVM and want to Interop with Java code, that doesn't mean it's idiomatic code.
Re: pigz: A parallel implementation of gzip for multi-core machines
#73Earlier quoted context omitted.
I didn't have the tar, I created it using: tar --use-compress-program="pigz -0" ...
I think you are a little confused here. Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression: tar -cf $directory.tar $directory You can also pipe the created archive to stdout instead of to a file if you want to: tar -c $directory | wc -c # send to a remo…
I know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure if pigz only uses multiple cores for the bzip algorithm, or if it somehow could speed up the packing as well). I'm pretty sure that disk is the bottleneck in most cases, but I wonder if that's not the case always, and a single core could become the bottleneck in very fast IO devices. I want to verify this and test it.
Re: pigz: A parallel implementation of gzip for multi-core machines
#74Earlier quoted context omitted.
zstd has --adapt: --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed man…
I really should have read the documentation! That feature looks awesome, but in a quick test it could only use about 50% of the available output bandwidth. My upload speed is 50 Mbps, but zstd could only send about 25 Mbps. Similarly, on a local speed test (SSD -> SSD), using a fixed compression level was much faster than --adapt.
My copy of that manual page has additional text:
"" note : at the time of this writing, --adapt can remain stuck at low speed when combined with multiple worker threads (>=2). ""
Leave wlog alone unless you're willing to store the value out of band and pass it in again during decompression.
hashLog, bigger number uses more memory to compress but is often faster.
chainLog smaller number compresses faster, but worse ratio.
In your use case monitoring general system utilization to identify bottlenecks might also help. My gut instinct is that you might already have hit a memory bandwidth limit for the platform, at which point REDUCING the hashLog until it fits within your intended performance budget might yield better bandwidth results. Reducing the chainLog value might have the same effect.
Re: pigz: A parallel implementation of gzip for multi-core machines
#75I had used pigz for a few years, now I've replace it with `xz -T0`
Re: pigz: A parallel implementation of gzip for multi-core machines
#76Earlier quoted context omitted.
pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block…
I'm a little confused by this. My copy of zstd has the option --format=gzip; does choosing this option end up using a different, slower compression algorithm?
zstandard can indeed handle standard format gzip files to create and decompress them. From the zstandard compilation options:
HAVE_ZLIB : zstd can compress and decompress files in .gz format. This is ordered through command --format=gzip. Alternatively, symlinks named gzip or gunzip will mimic intended behavior. .gz support is automatically enabled when zlib library is detected at build time. It's possible to disable .gz support, by setting HAVE_ZLIB=0. Example : make zstd HAVE_ZLIB=0 It's also possible to force compilation with zlib support, using HAVE_ZLIB=1. In which case, linking stage will fail if zlib library cannot be found. This is useful to prevent silent feature disabling.
Re: pigz: A parallel implementation of gzip for multi-core machines
#77Earlier quoted context omitted.
I think you are a little confused here. Tar is a standalone archive format, you can create tar archives with the tar command and not use any compression utility at all. Here is an example that creates an uncompressed tar archive directly with no compression: tar -cf $directory.tar $directory You can also pipe the created archive to stdout instead of to a file if you want to: tar -c $directory | wc -c # send to a remo…
I know about all that. My mistake was thinking that I could take advantage of the multicore nature of pigz even when no compression is being used (I'm not sure if pigz only uses multiple cores for the bzip algorithm, or if it somehow could speed up the packing as well). I'm pretty sure that disk is the bottleneck in most cases, but I wonder if that's not the case always, and a single core could become the bottleneck…
Oh yeah in that case tar itself is still creating the archive with a single thread, the compression gets applied "after" tar creates the archive almost exactly like in the pipeline in my comment.
By "after", I don't mean the whole archive is created before the compression part runs, but rather that tar can stream the the archive to stdout as it's being created. Tar needs to be able to do this since it was intended for tape drives, and seeking around on a tape drive is extremely slow.
It's actually pretty interesting that we still use tar so widely even though tape drives are not in common use! A new archive format that supports random access during reads (and writes maybe?) would be pretty interesting, but tar works pretty well so there isn't that much of a reason to create alternatives.
Re: pigz: A parallel implementation of gzip for multi-core machines
#78On Linux would it Just Work™ if you aliased pigz to gzip as a drop-in replacement?
I've done that without issue in the past. It's argument compatible so never seemed to be a problem
Re: pigz: A parallel implementation of gzip for multi-core machines
#79One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code chu…
Docker is indeed looking for a "unpigz" executable to use:
https://github.com/moby/moby/blob/c9d2b7df777b38f7239a882c27...
So interesting if they implemented and tested that and get only a marginal CRC speedup.
edit: someone here seems to observe a ~ doubling with unpigz vs zcat: https://unix.stackexchange.com/a/363739
Re: pigz: A parallel implementation of gzip for multi-core machines
#80One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.
pigz only parallelizes compression. Decompressing with pigz is single threaded, except perhaps a separate thread is used for crc calculation. A decade ago I implemented parallel decompresssion for pigz. This is used in Solaris kernel zone suspend and resume, which was the reason I did the work. I submitted a PR for it but madler never got around to reviewing and merging it. Since then there has been a lot of code chu…
I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.