131–140 of 197 posts
Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.
Earlier quoted context omitted.
> the effective rate hits 12 GiB/s I assume this is for decompressing multiple independent deflate streams in parallel? What's the throughput if you only have a single stream? I realise this is the unhappy-case for GPU acceleration, hence my question! (I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy)
> I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy You saw this, right? https://news.ycombinator.com/item?id=35915285
Earlier quoted context omitted.
I saw that zstd and brotli both suppport creating custom dictionaries but I couldn't find any tutorials showing how to do this. Perhaps you could share code?
Basically, `zstd --train ` will output a dictionary file, and then the `-D ` option when used for either compression or decompression will then use that dictionary first. You can also investigate "man zstd" or google "zstd --train" for more details. The directory for the training must consist of many small files each of which is an example artifact; if you want to split, say, a single log file into files of each line…
John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
The issue with pigz is that uncompressing doesn't really parallelize beyond a three stage read/uncompress/write pipeline. This is of course more of a problem of the gz format than pigz although last time I looked hacks are possible to parallelize decompression.
John Carmack just had a tweet today on this problem: >I started a tar with bzip command on a big directory, and it has been running for two days. Of course, it is only using 1.07 cores out of the 128 available. The Unix pipeline tool philosophy often isn’t aligned with parallel performance. https://twitter.com/ID_AA_Carmack/status/1656708636570271768...
Earlier quoted context omitted.
People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…
> No one ever tweaks that one setting even though they should That entirely depends on the use-case. Most people running FFMpeg do it as a once off thing - and if those people like me, when I rip a movie I want the highest quality and lowest size I can get, and I'm happy that the default sacrifices speed for quality and size. The processing can be slow because I'm doing it only once. If you're in the business of enco…
Earlier quoted context omitted.
> I've been thinking about some approaches to parallelize decompression of a single stream, it's not easy You saw this, right? https://news.ycombinator.com/item?id=35915285
There also is this: https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able to reach almost 20 GB/s decompression bandwidth.
I heard of pigz in the discussions following my interview of Yann Collet, creator of LZ4 and zstd. If you'll excuse the plug, here is the LZ4 story: Yann was bored and working as a project manager. So he started working on a game for his old HP 48 graphing calculator. Eventually, this hobby led him to revolutionize the field of data compression, releasing LZ4, ZStandard, and Finite State Entropy coders. His code ende…
I work in VMWare Fusion on a Mac, in a Mint guest OS, and zipping these huge instances for backup will take forever with a single core. Pigz punishes all 12 cores on my Mac mini and saves me a ton of time.
Earlier quoted context omitted.
People take performance for granted. Even within gzip (and similarly .png), you can set compression level to 4 (default is 6) and get ~15-20% faster performance at the cost of ~5% larger file sizes. No one ever tweaks that one setting even though they should, file sizes are a significantly smaller bottleneck than they were with MB hard drives and dial-up modems. If your justification for not serving up larger .png is…
Economics of scale come into effect as well. Gzip decompression speed is slightly better at higher levels as well. A one time higher cost of compression can pay off pretty quickly when you are decompressing it a lot of times, or serving it to enough people.
My gut feeling is that if you are pulling down data faster than 40 Megabits and have a CPU made within the past 7 years (possibly including mobile), you won't be bottlenecked by I/O generally speaking.