Hellishly Slow Level 13 Deflate Compression
kirill.korins.ky
Hellishly Slow Level 13 Deflate Compression
1–10 of 29 posts
Re: Hellishly Slow Level 13 Deflate Compression
#2I love it. So much in computers is trade offs and this was a fun read exploring it.
It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
Re: Hellishly Slow Level 13 Deflate Compression
#3I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themselves.
Re: Hellishly Slow Level 13 Deflate Compression
#4I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
[dead]
Re: Hellishly Slow Level 13 Deflate Compression
#5There is also zopfli and it's decadent ECT that allow for more extreme tradeoffs.
Re: Hellishly Slow Level 13 Deflate Compression
#6I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
I also wonder how brotli/lzma would compare here.
Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.
Re: Hellishly Slow Level 13 Deflate Compression
#7I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themsel…
To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).
Re: Hellishly Slow Level 13 Deflate Compression
#8So, what’s the effect on memory usage?
And for decompression, the effect on memory usage and timings?
Re: Hellishly Slow Level 13 Deflate Compression
#9It's interesting to see just how far Deflate can be taken, and to know that even after decades there is still some (admittedly tiny) room for improvement.
Optimal LZ is well-known, and so is static Huffman, but their combination creates some additional inefficiencies(opportunities).
...and of course it's written by someone with a Russian name, and has that characteristic style common to many other articles about data compression.
Re: Hellishly Slow Level 13 Deflate Compression
#10So, what’s the effect on memory usage? And for decompression, the effect on memory usage and timings?
For decompression, nothing changes because DEFLATE is asymmetric; compressor can spend however much time to optimize the compressed stream independently from decompressor.