Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming compression beats framed compression

bou.ke

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Streaming compression beats framed compression

#11

Post by yellow_lead »

duskwuff wrote: post #3

Before you get too excited, keep two things in mind: 1) Using a single compression context for the whole stream means you have to keep that context active on the client and server while the connection is active. This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) 2) Using a single context also means that you can't decompress…

> This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!)

It sounds like these contexts should be cleared when they reach a certain memory limit, or maybe reset periodically, i.e every N messages. Is there another way to manage the memory cost?

Re: Streaming compression beats framed compression

#12

Post by treyd »

yellow_lead wrote: post #11
duskwuff wrote: post #3

Before you get too excited, keep two things in mind: 1) Using a single compression context for the whole stream means you have to keep that context active on the client and server while the connection is active. This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) 2) Using a single context also means that you can't decompress…

> This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) It sounds like these contexts should be cleared when they reach a certain memory limit, or maybe reset periodically, i.e every N messages. Is there another way to manage the memory cost?

That's a misunderstanding. Compression algorithms are typically designed with a tunable state size paramter. The issue is if you have a large transfer that might have one side crash and resume, you need to have some way to persist the state to be able to pick up where you left off.

Re: Streaming compression beats framed compression

#14

Post by michaelt »

yellow_lead wrote: post #11
duskwuff wrote: post #3

Before you get too excited, keep two things in mind: 1) Using a single compression context for the whole stream means you have to keep that context active on the client and server while the connection is active. This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) 2) Using a single context also means that you can't decompress…

> This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) It sounds like these contexts should be cleared when they reach a certain memory limit, or maybe reset periodically, i.e every N messages. Is there another way to manage the memory cost?

LZ77 compression (a key part of gzip and zip compression) uses a 'sliding window' where the compressor can tell the decompressor 'repeat the n bytes that appeared in the output stream m bytes ago'. The most widely used implementation uses a 15 bit integer for m - so the decompressor never needs to look more than 32,768 bytes back in its output stream.

Many compression standards include memory limits, to guarantee compatibility, and the older the standard the lower that limit is likely to be. If the standards didn't dictate this stuff, DVD sellers could release a DVD that needed a 4MB decompression window, and it'd fail to play on players that only had 2MB of memory - setting a standard and following it avoids this happening.

Re: Streaming compression beats framed compression

#15

Post by skulk »

masklinn wrote: post #10

Surely that is obvious to anyone who has compared zip and tgz?

MUD clients and servers use MCCP which is essentially keeping a zlib stream open, adding text to it, and flushing it whenever something is received. I think this has been around since 2000.

https://tintin.mudhalla.net/protocols/mccp/

Post reply on HN