Hellishly Slow Level 13 Deflate Compression

Post by pella » Sat, Jun 27, 2026, 10:19 AM UTC

Retro_Dev wrote: post #17

pella wrote: post #16

OpenZL is the future: https://openzl.org/ "OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."

OpenZL is nice, but it's often less useful than you think - it requires that you know the structure of your data, and don't care about inspecting that data outside of your program. I've extracted one too many png files from a word document (by renaming .docx to .zip) to desire OpenZL everywhere... It might be better as a short-term "data in transit" compression than for long term storage.

  "OpenZL to offer 10% faster compression speed and 70% faster decompression speed compared to Zstandard level 1 on the Silesia corpus in our benchmarks."
  "OpenZL now ships its own LZ codec, exposed as ZL_GRAPH_LZ, and the serial profile in zli. It is still being actively developed to expand its feature set and improve performance on small inputs."

Post by chickenbig » Sat, Jun 27, 2026, 12:33 PM UTC

a_t48 wrote: post #3

jbosh wrote: post #2

I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?

zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themsel…

Post by mike_hock » Sat, Jun 27, 2026, 2:56 PM UTC

pella wrote: post #22

pella wrote: post #16

OpenZL is the future: https://openzl.org/ "OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."

The future may be ~ AI-assisted format detection + OpenZL (~ OpenZL-AI-LLM recognises the data structure, then guides OpenZL toward the best lossless compression path )

Post by pella » Sat, Jun 27, 2026, 3:17 PM UTC

pella wrote: post #22

pella wrote: post #16

OpenZL is the future: https://openzl.org/ "OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."

The future may be ~ AI-assisted format detection + OpenZL (~ OpenZL-AI-LLM recognises the data structure, then guides OpenZL toward the best lossless compression path )

  "The unreasonable effectiveness of our first foray into training leads us to believe that the graph model is uniquely  positioned to facilitate ML-guided generation of compressors. We are tempted to view this as “the next big thing” in production-scale compression. Whereas compression research has up to now eluded those without domain expertise, we believe the future of application-specific compressors will be unlocked via investment in automated learning methods."

Post by rini17 » Sun, Jun 28, 2026, 1:10 PM UTC

XorNot wrote: post #24

lstodd wrote: post #19

Earlier quoted context omitted.

RTG degrade no matter what you do with the resulting heat. It doesn't matter if you compress stuff or just let them cpus idle be.

That's the point: you're going to spend a lot more time compressing when you could've just been sending data. And you're eating into a limited overall power and weight budget to do rather then say, run the science on the probe.