Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hellishly Slow Level 13 Deflate Compression

kirill.korins.ky

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Hellishly Slow Level 13 Deflate Compression

#11

Post by a_t48 »

loeg wrote: post #7
a_t48 wrote: post #3

Earlier quoted context omitted.

zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themsel…

To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).

Whoops! You're right, and it's too late to edit.

Re: Hellishly Slow Level 13 Deflate Compression

#12

Post by blobbers »

As someone currently exploring grid searches of encodings + compressor combos, and currently looking at neural compressors that reduce size almost half that of a traditional compressor yet take order from ms -> minutes to operate in either direction, I appreciate a good compression post!

Re: Hellishly Slow Level 13 Deflate Compression

#14

Post by edflsafoiewq »

loeg wrote: post #7
a_t48 wrote: post #3

Earlier quoted context omitted.

zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themsel…

To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).

I think those are CLI options, not negative signs. Ie. you call zstd -3 for compression level 3.

Re: Hellishly Slow Level 13 Deflate Compression

#15

Post by Zenst »

jbosh wrote: post #2

I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?

Process-intensive, but higher compression has clear strategic value. Distant satellites such as Voyager, where bandwidth is severely limited, could transmit more data using such capabilities. Equally, for long-term archival storage, improved compression would allow far greater volumes of data to be preserved on durable, life-long media formats.

Re: Hellishly Slow Level 13 Deflate Compression

#16

Post by pella »

OpenZL is the future: https://openzl.org/

  "OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."

Re: Hellishly Slow Level 13 Deflate Compression

#17

Post by Retro_Dev »

pella wrote: post #16

OpenZL is the future: https://openzl.org/ "OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."

OpenZL is nice, but it's often less useful than you think - it requires that you know the structure of your data, and don't care about inspecting that data outside of your program. I've extracted one too many png files from a word document (by renaming .docx to .zip) to desire OpenZL everywhere... It might be better as a short-term "data in transit" compression than for long term storage.

Re: Hellishly Slow Level 13 Deflate Compression

#18

Post by XorNot »

Zenst wrote: post #15
jbosh wrote: post #2

I love it. So much in computers is trade offs and this was a fun read exploring it. It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?

Process-intensive, but higher compression has clear strategic value. Distant satellites such as Voyager, where bandwidth is severely limited, could transmit more data using such capabilities. Equally, for long-term archival storage, improved compression would allow far greater volumes of data to be preserved on durable, life-long media formats.

Distant space probes are power constrained though.

It's entirely possible the degradation of their RTG power sources would be more expensive doing the compression then just sending the data as is.

Re: Hellishly Slow Level 13 Deflate Compression

#19

Post by lstodd »

XorNot wrote: post #18
Zenst wrote: post #15

Earlier quoted context omitted.

Process-intensive, but higher compression has clear strategic value. Distant satellites such as Voyager, where bandwidth is severely limited, could transmit more data using such capabilities. Equally, for long-term archival storage, improved compression would allow far greater volumes of data to be preserved on durable, life-long media formats.

Distant space probes are power constrained though. It's entirely possible the degradation of their RTG power sources would be more expensive doing the compression then just sending the data as is.

RTG degrade no matter what you do with the resulting heat. It doesn't matter if you compress stuff or just let them cpus idle be.

Re: Hellishly Slow Level 13 Deflate Compression

#20

Post by masklinn »

lifthrasiir wrote: post #10
Someone wrote: post #8

So, what’s the effect on memory usage? And for decompression, the effect on memory usage and timings?

For decompression, nothing changes because DEFLATE is asymmetric; compressor can spend however much time to optimize the compressed stream independently from decompressor.

Deflate also has a fixed 32K window so even with indexes and parallelism there’s only so much you can blow up memory use.
Post reply on HN