Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chrome feature: Compression dictionary transport with Shared Brotli

chromestatus.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#21

Post by tyingq »

philsnow wrote: post #17
tyingq wrote: post #3

Makes me wonder if rsync-style differential download would be more generally useful than "shared dictionary for shared brotli". The implementation is letting you reference a previously downloaded artifact using a hash value, which is the same starting point.

Are you referring to courgette (which the Chrome team uses to make shipping patches/updates take far fewer bytes than previously)? https://www.chromium.org/developers/design-documents/softwar...

Yes, but exposed to everyone. This proposal settles on headers that get passed back and forth that surely some servers would adopt. Headers that established "previously downloaded thing" with a hash. It feels like the process could be exactly the same at the start, but end instead with some data to the client that it could use to do the http range requests. This proposal feels like it could be extended in a fairly easy way to support differential download.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#22

Post by __alexs »

jacobgorm wrote: post #10
__alexs wrote: post #6

Earlier quoted context omitted.

Zstd no longer comes with a patent grant which I guess makes it kind of hard to actually use for some people.

Zstd is already used in the Linux kernel, and I would assume that any patents covering Lempel-Ziv and Huffman coding would have expired long ago. The situation around ANS also seems to have been resolved, see https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#Tab... .

The Linux kernel is stuffed full of potentially patent encumbered code.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#24

Post by pimterry »

rasz wrote: post #23

>The client will store a hash of the uncompressed response >client will add ...request header as well as a sec-available-dictionary: loving it, another thing to track clients with

From the spec (https://github.com/WICG/compression-dictionary-transport#fin...):

> The existence of a dictionary is effectively a cookie for any requests that match it and should be treated as such:

> * Storage partitioning for dictionary resource metadata should be at least as restrictive as for cookies.

> * Dictionary entries (or at least the metadata) should be cleared any time cookies are cleared.

Given that, I think it offers no more client tracking possibilities than any other existing web tech (cookies, local storage, etc)..

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#25

Post by Scaevolus »

They already had a very similar feature. _15 years ago_ with SDCH that got removed in 2018-- https://chromestatus.com/feature/5763176272494592 https://daniel.haxx.se/blog/2008/09/09/shared-dictionary-com...

Presumably they have some reason to think that this will be useful now when it wasn't before?

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#26

Post by jauntywundrkind »

jacobgorm wrote: post #2

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. Is it a Google-NIH thing?

Brotli has two major advantages. First it encodes to the same well known gzip format, so rolling it out didn't require new clients. Second, it didn't require dictionary support.

Where-as if you look at Mozilla's standards-positions for zstd, they defer because it requires a compression dictionary to encode & Moz didn't want to have to "standardize" one specific dictionary. https://github.com/mozilla/standards-positions/issues/105

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#27

Post by pmarreck »

lifthrasiir wrote: post #9
jacobgorm wrote: post #2

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. Is it a Google-NIH thing?

They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both designed to be much more efficient than DEFLATE in both time and size axes, but otherwise have little similarities. Zstandard is notable in its use of ANS, while Brotli is probably the first mainstream format that prominently uses a 2nd-order context model wh…

Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#28

Post by pmarreck »

jacobgorm wrote: post #2

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. Is it a Google-NIH thing?

The Zstd guy is employed by Facebook, so I'd suspect it's at least in part a NIH thing.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#29

Post by pmarreck »

jauntywundrkind wrote: post #26
jacobgorm wrote: post #2

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. Is it a Google-NIH thing?

Brotli has two major advantages. First it encodes to the same well known gzip format, so rolling it out didn't require new clients. Second, it didn't require dictionary support. Where-as if you look at Mozilla's standards-positions for zstd, they defer because it requires a compression dictionary to encode & Moz didn't want to have to "standardize" one specific dictionary. https://github.com/mozilla/standards-positio…

> Moz didn't want to have to "standardize" one specific dictionary

You could have multiple sources of dictionary identified by something like a hash, and cached in the browser. For example you could have a general-purpose HTML dictionary, a general-purpose CSS dictionary and a general-purpose JS dictionary, have canonical/official versions of these hosted somewhere (say, on Github) and distributed wherever, and update them on a semi regular basis (at the cost of re-compression of the associated data); and in turn these could be overrided on perhaps a site by site basis with site-specific versions of these. You could basically associate an overridable but defaulted dictionary specification by MIME type (or none at all in the case of already-compressed data).

Every file would have to be sent with the hash of the dictionary that was used to start compressing it, which would then be retrieved and cached somehow.

I could see this resulting in some fairly significant byte savings, especially for retrieval of many smaller files.

I bet this would also be useful potentially for email clients/servers.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#30

Post by TonyTrapp »

pimterry wrote: post #24
rasz wrote: post #23

>The client will store a hash of the uncompressed response >client will add ...request header as well as a sec-available-dictionary: loving it, another thing to track clients with

From the spec ( https://github.com/WICG/compression-dictionary-transport#fin... ): > The existence of a dictionary is effectively a cookie for any requests that match it and should be treated as such: > * Storage partitioning for dictionary resource metadata should be at least as restrictive as for cookies. > * Dictionary entries (or at least the metadata) should be cleared any time cookies are cleared. Given that, I…

Maybe no more possibilities than regular cookies, but more stealthy for sure. Cookies are (somewhat) easily reviewable by the user through the web developer UI of most browsers; I doubt there will be any browser UI to managing compression dictionary.
Post reply on HN