Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#1

Post by DiabloD3 » Wed, May 06, 2015, 8:25 PM UTC

So, pretty much everyone agrees that LZ4 is a superior codec for on-the-fly compression of data, and works especially well with HTML and XML and JSON.

Yet, we still only allow gzip for HTTP compression. What gives? I use gzip -9 + advdef -z -4 -i 1000 + nginx's gzip_static for static resources; but with dynamic HTML, I'm stuck with gzip -1 (with anything more, compression tends to take longer than the data saved wrt a modern Internet connection (>10mbps)).

LZ4 would perform better here and require less CPU on the remote side (not a big thing on desktops and laptops, but a huge thing on cell phones and such), so why did Google and Mozilla and the IETF working group overlook this?

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#2

Post by Someone1234 » Wed, May 06, 2015, 10:27 PM UTC

In a one word answer: Mobile.

LZ4 might be better at compressing, but it has worse performance characteristics. But don't take my word for it:

http://catchchallenger.first-world.info//wiki/Quick_Benchmar...

As you can see, LZ4 uses 30x as much memory as gzip.

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#3

Post by DiabloD3 » Thu, May 07, 2015, 12:41 AM UTC

Someone1234 wrote: post #2

In a one word answer: Mobile. LZ4 might be better at compressing, but it has worse performance characteristics. But don't take my word for it: http://catchchallenger.first-world.info//wiki/Quick_Benchmar... As you can see, LZ4 uses 30x as much memory as gzip.

Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly.

And that doesn't answer my question: I'm okay with gzip also being allowed, but the whole entire reason we allow multiple compressors is so the web browser and the web server can negotiate what features are supported.

Low memory HTTP clients, yes, can continue to use gzip. But the rest of us, why are we not using LZ4? I mean, hell, there is no reason why HTTP 1.0 and 1.1 clients can't also support LZ4 as well: they just have to negotiate for the feature.

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#4

Post by gumby » Thu, May 07, 2015, 1:36 AM UTC

DiabloD3 wrote: post #3

Someone1234 wrote: post #2

In a one word answer: Mobile. LZ4 might be better at compressing, but it has worse performance characteristics. But don't take my word for it: http://catchchallenger.first-world.info//wiki/Quick_Benchmar... As you can see, LZ4 uses 30x as much memory as gzip.

Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly. And that doesn't answer my question: I'm okay with gzip also being allowed, but the whole entire reason we allow multiple compressors is so the web browser and the web server can negotiate what features are supported. Low memory HTTP clients, yes, can continue to use gzip. But the rest of us, why are we not u…

> Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly.

Memory access costs power and time (which = power too).

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#5

Post by SamReidHughes » Thu, May 07, 2015, 1:47 AM UTC

That makes more work for people to do, makes a bigger attack surface for hackers.

Also 12MB of memory used on compression (as seen in Someone1234's link) would be pretty bad for a web server.

Edit: Also, do phones have trouble keeping up with gzip decompression? Isn't it layout that's the limiting factor?

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#6

Post by tandemstrong » Thu, May 07, 2015, 5:52 AM UTC

Someone1234 wrote: post #2

In a one word answer: Mobile. LZ4 might be better at compressing, but it has worse performance characteristics. But don't take my word for it: http://catchchallenger.first-world.info//wiki/Quick_Benchmar... As you can see, LZ4 uses 30x as much memory as gzip.

This benchmark is a fairly bad reference.

It measures memory usage from the lz4 command line utility, without mentioning it. This is completely different from lz4 algorithm.

Effectively, it measures input / output buffers, which are external to lz4. Plus it uses highest settings for such buffers, also without mentioning it. Using -B4 setting, for example, would reduce I/O buffer memory by a 50X factor.

lz4 itself uses only 16KB for compression.

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#7

Post by stephenr » Thu, May 07, 2015, 6:38 AM UTC

DiabloD3 wrote: post #3

Someone1234 wrote: post #2

In a one word answer: Mobile. LZ4 might be better at compressing, but it has worse performance characteristics. But don't take my word for it: http://catchchallenger.first-world.info//wiki/Quick_Benchmar... As you can see, LZ4 uses 30x as much memory as gzip.

Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly. And that doesn't answer my question: I'm okay with gzip also being allowed, but the whole entire reason we allow multiple compressors is so the web browser and the web server can negotiate what features are supported. Low memory HTTP clients, yes, can continue to use gzip. But the rest of us, why are we not u…

I don't want my browser on my phone to require a gb of memory thank you very much.

It's bad enough that desktop browsers like chrome are complete memory hogs without stuff like this, I hate to imagine what a phone browser would be like if "fuck it, modern phones have heaps of resources" was the approach to all decisions.

Next you'll be suggesting a compression algorithm that only works anywhere near realtime when it has 8x1ghz+ cores to spread the work over.

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#8

Post by DiabloD3 » Thu, May 07, 2015, 8:03 AM UTC

gumby wrote: post #4

DiabloD3 wrote: post #3

Earlier quoted context omitted.

Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly. And that doesn't answer my question: I'm okay with gzip also being allowed, but the whole entire reason we allow multiple compressors is so the web browser and the web server can negotiate what features are supported. Low memory HTTP clients, yes, can continue to use gzip. But the rest of us, why are we not u…

> Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly. Memory access costs power and time (which = power too).

Yes it does, but does it cost more than the CPU power used by gzip vs the memory access cost of lz4? I suspect the CPU cost is more by at least a magnitude (although I can't measure it, but I bet Google has frameworks for Android to).

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#9

Post by DiabloD3 » Thu, May 07, 2015, 8:05 AM UTC

stephenr wrote: post #7

DiabloD3 wrote: post #3

Earlier quoted context omitted.

Memory usage isn't that big of a concern anymore, not when phones ship with 1 or more GB of memory regularly. And that doesn't answer my question: I'm okay with gzip also being allowed, but the whole entire reason we allow multiple compressors is so the web browser and the web server can negotiate what features are supported. Low memory HTTP clients, yes, can continue to use gzip. But the rest of us, why are we not u…

I don't want my browser on my phone to require a gb of memory thank you very much. It's bad enough that desktop browsers like chrome are complete memory hogs without stuff like this, I hate to imagine what a phone browser would be like if "fuck it, modern phones have heaps of resources" was the approach to all decisions. Next you'll be suggesting a compression algorithm that only works anywhere near realtime when it…

Your phone won't require it. It will temporarily use slightly more memory (12MB is very little on modern Android phones), and it would decrease the time required to render the page (due to less CPU power required to compress then decompress over gzip -1).

Re: Ask HN: LZ4 was not added as an optional compressor in HTTP/2, why gzip only?

#10

Post by DiabloD3 » Thu, May 07, 2015, 8:08 AM UTC

SamReidHughes wrote: post #5

That makes more work for people to do, makes a bigger attack surface for hackers. Also 12MB of memory used on compression (as seen in Someone1234's link) would be pretty bad for a web server. Edit: Also, do phones have trouble keeping up with gzip decompression? Isn't it layout that's the limiting factor?

Possibly not bigger attack surface. lz4 is already used in a lot of mission/security critical areas, and they have audited many lz4 implementations for security and have found and fixed bugs.

For example, the Linux kernel can use LZ4, corporate VPN solutions like OpenVPN can use LZ4, and many other things can use or require LZ4.