51–60 of 107 posts
Earlier quoted context omitted.
But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
Golang has panic / recover / defer which are functionally similar to exceptions. It's actually a fun exercise to implement a pseudo-syntax for try/catch/finally in terms of those primitives.
Earlier quoted context omitted.
Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream. This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order. It is the very same property of the format that gzip's own --rsyncable optio…
Given that, why wouldn't this just be upstreamed into gzip? If it's a clean, simple solution that's just expanding the use of a technique that's already in the core binary?
There are multiple implementations of zlib that are faster than the one that ships with GNU gzip, and yet they haven't been incorporated.
There are also just better algorithms if compatibility with gzip isn't needed. zstd, for example, supports parallel compression, and is both faster and compresses better than gzip.
I chuckled at the name, since out-of-order results are a typical output of parallelization. Kudos.
That’d be a pretty funny compression algorithm. You listen to a .mpfoo file, and you’ll hear the whole song, we promise!
Earlier quoted context omitted.
I used tar --use-compress-program="pigz" to create the tar out of billions of files
But what’s confusing everyone is that tar cf - will create the tar without any external compression program needed.
The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.
But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
All three of those languages actually have exceptions, they just don't encourage catching exceptions as a normal way of error handling
Also, while the trend now seems to be for newer languages to encourage use of things like result types, one of the main reasons for that is that in current languages it is easier to show that functions can potentially failure in the type system using result types rather than exceptions.
Otherwise, there isn't necessarily inherently a strong reason to prefer one or the other, and it's possible that future languages will go back to exceptions but have a way to express that in the type system using effects, etc.
One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also copy to cloud storage in parallel! E.g.:
pigz input.fastq -c | azcopy copy --from-to PipeBlob "https://myaccountname.blob.core.windows.net/inputs/input.fastq.gz?..."
However, if your tools support zstd, then it's more efficient to use that instead. Try the "zstd -T0" option or the "pzstd" tool for even higher throughputs but with same minor caveats.
PS: In case anyone here is working on the above tools, I have a small request! What would be awesome is to automatically tune the compression ratio to match the available output bandwidth. With the '-c' output option, this is easy: just keep increasing the compression level by one notch whenever the output buffer is full, and reduce it by one level whenever the output buffer is empty. This will automatically tune the system to get the maximum total throughput given the available CPU performance and network bandwidth.
Earlier quoted context omitted.
Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream. This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order. It is the very same property of the format that gzip's own --rsyncable optio…
Given that, why wouldn't this just be upstreamed into gzip? If it's a clean, simple solution that's just expanding the use of a technique that's already in the core binary?
I suspect to keep the standard gzip as simple, small, and stable, as possible. It does the job, does it well enough, with minimal dependencies, has done for many years, and can do so in a wide array of systems including very small environments (in part due to the minimal dependencies).
Core tools like that typically don't get major updates, just security & stability patches as needed and maybe the occasional safe & easy change for performance reasons or to widen the number of supported environments.
Funny this comes up again so soon after I needed it! I recently did a proof-of-concept related to bioinformatics (gene assembly, etc...), and one quirk of that space is that they work with enormous text files. Think tens of gigabytes being a "normal" size. Just compressing and copying these around is a pain. One trick I discovered is that tools like pigz can be used to both accelerate the compression step and also co…
--adapt[=min=#,max=#]
zstd will dynamically adapt compression level to perceived I/O conditions. Compression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied
min and max levels. The feature works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed manu‐
ally, see wlog). Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible.I use this all the time. It's a big time saver on multi-core machines (which is pretty much every desktop made in the past 20 years). It's available in all the repos, but not included by default (at least in Ubuntu/Mint). It is most useful for compressing disk images on-the-fly while backing them up to network storage. It's usually a good idea to zero unused space first: (unprivileged commands follow) dd if=/dev/zero…
Alternative way of zeroing unused space without consuming all disk space: https://manpages.ubuntu.com/manpages/trusty/man8/zerofree.8....
The zerofree command looks useful, but I don't know how portable it is. The dd method works across many platforms (such as AIX).