Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chrome feature: Compression dictionary transport with Shared Brotli

chromestatus.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#31

Post by lifthrasiir »

pmarreck wrote: post #27
lifthrasiir wrote: post #9

Earlier quoted context omitted.

They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both designed to be much more efficient than DEFLATE in both time and size axes, but otherwise have little similarities. Zstandard is notable in its use of ANS, while Brotli is probably the first mainstream format that prominently uses a 2nd-order context model wh…

Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?

Not even close, zlib had one. (Search for "FDICT" from RFC 1950.) What Zstandard did was a CLI to automatically generate a good enough dictionary from sample inputs.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#32

Post by masklinn »

jauntywundrkind wrote: post #26
jacobgorm wrote: post #2

I never really understood with Google keeps pushing Brotli. LZ4 and ZStd both predate it by a couple of years, and seem to offer superior performance overall. Is it a Google-NIH thing?

Brotli has two major advantages. First it encodes to the same well known gzip format, so rolling it out didn't require new clients. Second, it didn't require dictionary support. Where-as if you look at Mozilla's standards-positions for zstd, they defer because it requires a compression dictionary to encode & Moz didn't want to have to "standardize" one specific dictionary. https://github.com/mozilla/standards-positio…

> Brotli has two major advantages. First it encodes to the same well known gzip format

It does not. You’re thinking about zopfli. Brotli is an alternative to but not compatible with gzip.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#33

Post by pmarreck »

lifthrasiir wrote: post #31
pmarreck wrote: post #27

Earlier quoted context omitted.

Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?

Not even close, zlib had one. (Search for "FDICT" from RFC 1950.) What Zstandard did was a CLI to automatically generate a good enough dictionary from sample inputs.

      FDICT (Preset dictionary)
         If FDICT is set, a DICT dictionary identifier is present
         immediately after the FLG byte. The dictionary is a sequence of
         bytes which are initially fed to the compressor without
         producing any compressed output. DICT is the Adler-32 checksum
         of this sequence of bytes (see the definition of ADLER32
         below).  The decompressor can use this identifier to determine
         which dictionary has been used by the compressor.
Well, wow. I have to wonder why this wasn't more utilized, then. There are a ton of contexts (columnar data in databases, for example) where shared-dictionary-based compression might have helped a ton before now.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#34

Post by qingcharles »

Scaevolus wrote: post #25

They already had a very similar feature. _15 years ago_ with SDCH that got removed in 2018-- https://chromestatus.com/feature/5763176272494592 https://daniel.haxx.se/blog/2008/09/09/shared-dictionary-com... Presumably they have some reason to think that this will be useful now when it wasn't before?

Talked about here:

https://github.com/WICG/compression-dictionary-transport

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#35

Post by out_of_protocol »

rasz wrote: post #23

>The client will store a hash of the uncompressed response >client will add ...request header as well as a sec-available-dictionary: loving it, another thing to track clients with

should be w3c dicts only and number of dicts and versions should be heavily limited. well, something like js,css,html/xml,json,common-web and new version every 5 years i think good enough for everyone

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#36

Post by pornel »

tyingq wrote: post #3

Makes me wonder if rsync-style differential download would be more generally useful than "shared dictionary for shared brotli". The implementation is letting you reference a previously downloaded artifact using a hash value, which is the same starting point.

It would be a dud. rsync-style download needs to prepare and upload significant amount of data to describe what the client already has. Hashes don't compress, and you want granular ones to reuse as much as possible.

Web is more sensitive to latency than bandwidth, and upload is more restricted than download. So rsync hits all the worst spots, for the least gain. The data that diffs nicely, like HTML, is already tiny and compresses very well. The large files, like videos, don't diff well at all.

There is a way to make rsync-style differential download really quick to request: the client and server need to agree on diffing only from a specific shared dataset… like a dictionary! And you can replace hashes with smaller variable-length integers.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#37

Post by jauntywundrkind »

pmarreck wrote: post #29
jauntywundrkind wrote: post #26

Earlier quoted context omitted.

Brotli has two major advantages. First it encodes to the same well known gzip format, so rolling it out didn't require new clients. Second, it didn't require dictionary support. Where-as if you look at Mozilla's standards-positions for zstd, they defer because it requires a compression dictionary to encode & Moz didn't want to have to "standardize" one specific dictionary. https://github.com/mozilla/standards-positio…

> Moz didn't want to have to "standardize" one specific dictionary You could have multiple sources of dictionary identified by something like a hash, and cached in the browser. For example you could have a general-purpose HTML dictionary, a general-purpose CSS dictionary and a general-purpose JS dictionary, have canonical/official versions of these hosted somewhere (say, on Github) and distributed wherever, and updat…

> You could have multiple sources of dictionary identified by something like a hash, and cached in the browser.

The browser actually has a very advanced implementation of something like this! They're called urls. The work here is to make it possible to identify dictionaries & retrieve & cache them in the browsers.

Mozilla I think smartly avoided creating a spec that requires some variety of centralized governance of what dictionaries to use, like you describe. Whatever the intent of such a community, it creates biases & will have a hard time figuring out what sets of dictionaries are worth creating, for what intents or code bases, and adapting or not as it discovers underserved code bases.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#38

Post by tyingq »

pornel wrote: post #36
tyingq wrote: post #3

Makes me wonder if rsync-style differential download would be more generally useful than "shared dictionary for shared brotli". The implementation is letting you reference a previously downloaded artifact using a hash value, which is the same starting point.

It would be a dud. rsync-style download needs to prepare and upload significant amount of data to describe what the client already has. Hashes don't compress, and you want granular ones to reuse as much as possible. Web is more sensitive to latency than bandwidth, and upload is more restricted than download. So rsync hits all the worst spots, for the least gain. The data that diffs nicely, like HTML, is already tiny…

Not rsync style in the sense of whole directory trees. Differential range-request download of the subsequent version of some specific file. Because this scheme means client and server already know a prior version via the hash.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#39

Post by philsnow »

tyingq wrote: post #21
philsnow wrote: post #17

Earlier quoted context omitted.

Are you referring to courgette (which the Chrome team uses to make shipping patches/updates take far fewer bytes than previously)? https://www.chromium.org/developers/design-documents/softwar...

Yes, but exposed to everyone. This proposal settles on headers that get passed back and forth that surely some servers would adopt. Headers that established "previously downloaded thing" with a hash. It feels like the process could be exactly the same at the start, but end instead with some data to the client that it could use to do the http range requests. This proposal feels like it could be extended in a fairly ea…

Interesting! Privacy concerns aside for the moment, I would guess that there is some "optimal" compression dictionary over some transferred data set, and that Google/Chrome of all people would be in a position to calculate it and distribute it along with Chrome.

Then Google could ship that dictionary to all (Chrome) browsers and update it once a month, and ship the same dictionary to anybody who wants to use it on the server side. Browsers can indicate which shared dictionaries they have preloaded and if there's overlap with which dictionaries the server has, the server can just choose one and use that to compress the stream (which seems kind of like ciphersuite negotiation). Compression should be faster and use way less memory if the sender can assume that the receiver already has the same dictionary, and if there's any problem, both sides can just fall back to on-the-fly compression with inlined dictionary like we've always done.

There are almost certainly different optimal dictionaries for different locales / countries, each of which could be calculated and distributed in the same way. Even if the 'wrong' dictionary gets used for a given request, it's (probably) not catastrophic.

I guess it might be possible for an attacker to indicate their client only has one dictionary and request pages that they know are disadvantageous for the server to have to compress with that dictionary. Even then, server-side heuristics can account for a lower-than-expected compression ratio and, again, fall back to on-the-fly dictionary calculation.

Re: Chrome feature: Compression dictionary transport with Shared Brotli

#40

Post by oynqr »

out_of_protocol wrote: post #8
oynqr wrote: post #5

Earlier quoted context omitted.

Brotli usually compresses plain text better than ZSTD.

Both with dictionaries? Both without? Any links to benchmarks?

Brotli comes with a dictionary by default, which is optimized for plaintext use cases. Once you give ZSTD a similar dictionary, its ratios are going to be similar/better.

I won't provide random links, because those can't replace testing on your own dataset.

Post reply on HN