41–50 of 107 posts
Earlier quoted context omitted.
bzip2 is very very slow though. Some types of data compress quite well with bzip, but if high compression is needed, xz is usually as good or better and natively has multithreading available. For everything else, there's zstd (also natively multithread)
Interesting https://docs.rs/zstd/latest/zstd/stream/write/struct.Encoder...
(unprivileged commands follow)
dd if=/dev/zero of=~/zeros bs=1M; sync; rm ~/zeros
Compressing on the fly can be slower than your network bandwidth depending on your network speed, your processor(s) speed, and the compression level, so you typically tune the compression level (because the other two variables are not so easy to change). Example backup:
(privileged commands follow)
pv (Note that on slower systems the ssh encryption can also slow things down.)
Some sharp people may notice that it's not necessarily a good idea to back up a live system this way because the filesystem is changing while the system runs. It's usually just fine on an unloaded system that uses a journaling filesystem.
Earlier quoted context omitted.
What is xzip? are you talking about xz?
yes, xz section 3.6 here https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Mar... https://en.wikipedia.org/wiki/XZ_Utils
I generally use lzip for data that is important to me.
Earlier quoted context omitted.
pigz has the advantage of producing output that can be read by standard gzip processing tools (including, of course, gzip/gunzip), which are available by default on just about every OS out there so you get the faster archive creation speed without adding requirements to those who might be accessing the results later. It works because gzip streams can be tracked together as a single stream, at the start of each block…
yes, one consideration is whether you're creating archives for your own later use, or internal use where you also have zstandard and xz handling tools. Or to send somewhere else for wider use on unknown platforms.
One interesting trivia is that since ~2020 Docker will transparently use pigz for decompressing container image layers if it's available on the host. This was a nice speedup for us, since we use large container images and automatic scaling for incoming traffic surges.
The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.
But why? Most modern languages try to get rid of exceptions (Go, Kotlin, Rust).
Earlier quoted context omitted.
I also thought the name was clever, but your comment made it even more interesting. Also, my first thought was, "is this safe to use?", I heard of gzip vulnerabilities before, but a parallel implementation sounds a lot easier to get wrong.
Gzip streams support dictionary resets which means you can concatenate individually commuters blocks together to make a while stream. This is what pigz is doing: shooting the input into blocks, spreading the compression of these blocks over different threads so multiple cores can be used, then joining the results together in the right order. It is the very same property of the format that gzip's own --rsyncable optio…
I use this all the time. It's a big time saver on multi-core machines (which is pretty much every desktop made in the past 20 years). It's available in all the repos, but not included by default (at least in Ubuntu/Mint). It is most useful for compressing disk images on-the-fly while backing them up to network storage. It's usually a good idea to zero unused space first: (unprivileged commands follow) dd if=/dev/zero…
We used this to great effect at Facebook for MySQL backups in the early 2010s. The backup hosts had far more CPU than needed so it was a very nice speed-up over gzip. Eventually we switched to zstd, of course, but pigz never failed us.