91–100 of 197 posts
I'm starting to get sick of those cartoon project names. Not sure what the alternative would be but it's increasingly rubbing me the wrong way.
Also - you'll never forget it.
Earlier quoted context omitted.
Unix generally favors processes over threads, at least the old school Unix. Threads are a more recent innovation. The old approach was that programs don't need internal parallelism because you can get it by just piping stuff and relying on the kernel's buffering to keep multiple processes busy. Eg, tar is running on one core dealing with the filesystem, gzip is running on another core compressing stuff. In the early…
I agree with most of these points except blaming this on processes vs threads. The only difference is all memory being shared by default, vs explicitly deciding what memory to share. With all the emphasis on memory safety on HN you think this point would be appreciated.
If you are interested in optimizing parallel decompression and you happen to have a suitable NVIDIA GPU, GDeflate [1] is interesting. The target market for this is PC games using DirectStorage to quickly load game assets. The graph in [1] shows DirectStorage maxing out the throughput of a PCIe Gen 3 drive at about 3 GiB/s when compression is not used. When GPU GDeflate is used, the effective rate hits 12 GiB/s. If yo…
Containerd will utilize unpigz if it’s on your PATH, thank me later: https://github.com/containerd/containerd/blob/main/archive/c...
Earlier quoted context omitted.
Unix generally favors processes over threads, at least the old school Unix. Threads are a more recent innovation. The old approach was that programs don't need internal parallelism because you can get it by just piping stuff and relying on the kernel's buffering to keep multiple processes busy. Eg, tar is running on one core dealing with the filesystem, gzip is running on another core compressing stuff. In the early…
“Recent” as in the 80s or 90s, sure. Threads are older than unix was when threads were introduced.
Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz. Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way. In 2023, friends don’t let friends use gzip.
`tar czf` is a lot easier to remember than `tar -I zstd cf`
Earlier quoted context omitted.
> as well explored as data compression What's well explored is compression rate, where indeed it's difficult to improve, and true innovations, like arithmetic coding, are rare. Compressing speed on the other hand it's not very interesting to academics, it's more of an engineering problem. And there is plenty of work to do here, starting with stuff as simple as multi-threading and SIMD. ZLib and ZStandard are probably…
> What's well explored is compression rate not performance Exactly! And this seems like a shame to me with something burning so many cpu cycles. > true innovations, like arithmetic coding, are rare. Yeah, Yann tried to explain arithmetic coding to me, but I didn't get it.
Consider this example: you want to transmit two completely random variables, both of which can have 5 states. The obvious way is to concatenate two bit fields of size ceil(log2(5)), so 3+3 = 6 bits.
But alternatively, you can count the total number of states possible for both variables together, 5*5 = 25 and encode it as a single integer of size ceil(log2(25)) = 5, so both variables can be stored with just 5 bits.
So we arrive at the idea that there can be a fractional number of bits, which we often round up to the nearest integer for simplicity (or, in practice, the nearest multiple of 8 since most protocols are based on bytes).
The other part is just assigning shorter sequences of bits to more common symbols, except, of course unlike in Huffman coding, our symbols can have a fractional number of bits. This allows them to match the actual symbol probabilities more closely. If your data is highly repetitive, you can fit dozens of (common) symbols per bit.
The coolest part IMO is how easy it is to plug in custom models for symbol probabilities. Usually a simple counter for each symbol is enough, but you can go crazy and start predicting the next symbol based on previous ones.
Earlier quoted context omitted.
For many small files, compressing first will compress worse, because each file has its own dictionary; compressing last means you can take advantage of similarities in files to improve compression ratio. Compressing first can also be slower if the average file size is smaller than the block size, because the main thread cannot queue new jobs as fast as cores complete them (this happens e.g. with 7zip at fast compress…
Zstandard has a dictionary functionality which allows you to pre-train on sample data to achieve higher compression ratios and faster compression on large numbers of small files.