Streaming compression beats framed compression
1–10 of 17 posts
Re: Streaming compression beats framed compression
#2Does streaming compression work if some packets are lost or arrive in a different order? Seems like the compression context may end up different on the encoding/decoding side.. or is that handled somehow?
Re: Streaming compression beats framed compression
#3Before you get too excited, keep two things in mind:
1) Using a single compression context for the whole stream means you have to keep that context active on the client and server while the connection is active. This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!)
2) Using a single context also means that you can't decompress one frame without having read the whole stream that led up to that. This prevents some possible useful optimizations if you're "fanning out" messages to many recipients - if you're compressing each message individually, you can compress it once and send the same compressed message to every recipient.
Re: Streaming compression beats framed compression
#4Does streaming compression work if some packets are lost or arrive in a different order? Seems like the compression context may end up different on the encoding/decoding side.. or is that handled somehow?
It sounds as though the data is being transferred over HTTP, so packet loss/reordering is all handled by TCP.
Re: Streaming compression beats framed compression
#5Does streaming compression work if some packets are lost or arrive in a different order? Seems like the compression context may end up different on the encoding/decoding side.. or is that handled somehow?
I think the underlying protocol would have to guarantee in order delivery - either via tcp (for http1, 2, or spdy), or in http3, within a single stream.
Re: Streaming compression beats framed compression
#6Before you get too excited, keep two things in mind: 1) Using a single compression context for the whole stream means you have to keep that context active on the client and server while the connection is active. This may have a nontrivial memory cost, especially at high compression levels. (Don't set the compression window any larger than it needs to be!) 2) Using a single context also means that you can't decompress…
The analogy to h264 in the original post is very relevant. You can fix some of the downsides by using the equivalent of keyframes, basically. Still a longer context than a single message but able to be broken up for recovery or etc.
Re: Streaming compression beats framed compression
#7When I worked at Microsoft years ago, me and my team (a developer and a tester) built a high volume log collector.
We used a streaming compression format that was originally designed for IBM tape drives.
It was fast as hell and worked really well, and was gentle on CPU and it was easy to control memory usage.
In the early 2000s on a modest 2-proc AMD64 machine we ran out of fast Ethernet way before we felt CPU pressure.
We got hit by the SOAP mafia during Longhorn; we couldn’t convince the web services to adopt it; instead they made us enshittify our “2 bytes length, 2 bytes msgtype, structs-on-the-wire” speed demon with their XML crap.
Re: Streaming compression beats framed compression
#8Does streaming compression work if some packets are lost or arrive in a different order? Seems like the compression context may end up different on the encoding/decoding side.. or is that handled somehow?
WebSockets [1] run over TCP, and the messages are ordered.
There is RFC 9220 [2] that makes WebSockets go over QUIC (which is UDP-based). But that's still expected to expose a stream of bytes to the WebSocket, which still keeps the ordering guarantee.
Re: Streaming compression beats framed compression
#9Does streaming compression work if some packets are lost or arrive in a different order? Seems like the compression context may end up different on the encoding/decoding side.. or is that handled somehow?
It sounds as though the data is being transferred over HTTP, so packet loss/reordering is all handled by TCP.
Yes, or by http3's in order guarantees on the individual streams (as http3 is udp)
Re: Streaming compression beats framed compression
#10Surely that is obvious to anyone who has compared zip and tgz?