Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression efficiency with shared dictionaries in Chrome

developer.chrome.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Compression efficiency with shared dictionaries in Chrome

#71

Post by patrickmeenan »

jauntywundrkind wrote: post #30

The Request For Position on Mozilla Zstd Support (2018) has a ton of interesting discussion on dictionaries. https://github.com/mozilla/standards-positions/issues/105 The original proposal for Zstd was to use a predefined stastically generated dictionary. Mozilla rejected the proposal for that. But there's a lot of great discussion on what Zstd can do, whic.h is astoundingly flexible & powerful. There's discussion on…

The brotli repo on github has a dictionary generator: https://github.com/google/brotli/blob/master/research/dictio...

I have a hosted version of it on https://use-as-dictionary.com/ to make it easier to experiment with.

Re: Compression efficiency with shared dictionaries in Chrome

#72

Post by patrickmeenan »

tsss wrote: post #60

This _screams_ sidechannel attack.

How so? SDCH had sidechannel issues which is part of why it was unshipped. I don't know that someone won't find a way to attack it but the CORS requirement already requires that the dictionary and compressed-resource be readable and the dictionary has to be same-origin as the resources that it compresses.

Combined they mitigate the known dictionary-specific attack vectors.

Re: Compression efficiency with shared dictionaries in Chrome

#73

Post by patrickmeenan »

IshKebab wrote: post #47

Ah damn I thought this was going to be available to JavaScript. Would be amazing for one use case I have (an HTML page containing inline logs from a load of commands, many of which are substantially similar).

Maybe eventually (as a different spec). We've talked about wanting to support it in the DecompressionStream API or something similar at some point.

If you need it to be able to do compression though then it might be a harder sell since the browser doesn't ship with the compression code for zstd or brotli and would have to justify adding it.

Re: Compression efficiency with shared dictionaries in Chrome

#74

Post by miohat »

EE84M3i wrote: post #62
lukevp wrote: post #61

This seems so ludicrous to me when all we really need is a way to share a resource reference across sites. Like “I need react 18.1 on this page, and the SHA should be abcdefghi “. If you don’t have it, I can give it to you from my server, or you can follow this link to a CDN, but the resource itself can be deduplicated based on the hashed contents instead of the URI. Why isn’t this a thing when basically everything u…

Privacy issues. You can use the presence of an item in the cache to correlate visits between sites.

LocalCDN can hijack requests for common static components

Re: Compression efficiency with shared dictionaries in Chrome

#75

Post by burnhamup »

Wingy wrote: post #55
hinkley wrote: post #46

Earlier quoted context omitted.

This is something game devs have been doing for decades. If you want to delta 1.0 to 1.1 that’s server side work you do once at deployment or build time, not on every request.

What happens when you release 1.2 and someone who has 1.0 visits? Do you generate a delta for every past version at build time?

The patch system I worked with generated signatures of each build. The signature had the hash of each block of the build. The client has the signature for their version (1.0) and they download the signature of the new version (1.2) and diff the two. Then they download each block that has changed.

I think it was the `electron-updater` for my electron app, but I don't quite remember now.

Re: Compression efficiency with shared dictionaries in Chrome

#76

Post by hinkley »

pyrolistical wrote: post #64
hinkley wrote: post #57

Earlier quoted context omitted.

And here’s one tuned for react, and one for svelte…

That wouldn’t make sense as it would be the user agent (aka your browser) that implements these shared dictionaries and they wouldn’t be able to add non-standard shared dictionaries for libs like react. If they could do that then they might as well preload the cache with all common libs like react from well known cdn urls.

Committee decided set of dictionaries.

I never cared for react, but I know beyond a doubt that someone influential will ask for a dictionary tuned for it.

Re: Compression efficiency with shared dictionaries in Chrome

#77

Post by IshKebab »

patrickmeenan wrote: post #73
IshKebab wrote: post #47

Ah damn I thought this was going to be available to JavaScript. Would be amazing for one use case I have (an HTML page containing inline logs from a load of commands, many of which are substantially similar).

Maybe eventually (as a different spec). We've talked about wanting to support it in the DecompressionStream API or something similar at some point. If you need it to be able to do compression though then it might be a harder sell since the browser doesn't ship with the compression code for zstd or brotli and would have to justify adding it.

Yeah just decompression through `DecompressionStream` is exactly what I'd like.

Re: Compression efficiency with shared dictionaries in Chrome

#78

Post by kevinventullo »

ColonelPhantom wrote: post #63
lukevp wrote: post #61

This seems so ludicrous to me when all we really need is a way to share a resource reference across sites. Like “I need react 18.1 on this page, and the SHA should be abcdefghi “. If you don’t have it, I can give it to you from my server, or you can follow this link to a CDN, but the resource itself can be deduplicated based on the hashed contents instead of the URI. Why isn’t this a thing when basically everything u…

One potential issue is tracking. By sharing caches across websites it becomes possible to use timing attacks to track different users. This is why browsers are working to isolate caches per site: https://developer.chrome.com/blog/http-cache-partitioning

Couldn’t a dedicated actor use IP address or other fingerprinting techniques to do the same thing more easily?

Re: Compression efficiency with shared dictionaries in Chrome

#79

Post by netol »

patrickmeenan wrote: post #70
netol wrote: post #43

The part I'm missing is how these dictionaries are created. Can I use the homepage to create my dictionary, so all other pages that share html are better efficiently compressed? How?

For a delta update of one version of a resource to the next, the resource itself is the dictionary (i.e. JS file). For stand-alone dictionaries, the brotli code on github has a dictionary_generator that you can use to generate a dictionary. You give it a dictionary size and a bunch of input files and it will generate one. I have a version of it hosted on https://use-as-dictionary.com/ that you can pass up to 100 URLs…

Cool, thanks!

Re: Compression efficiency with shared dictionaries in Chrome

#80

Post by jgrahamc »

lynguist wrote: post #58
jgrahamc wrote: post #4

The very first project I worked on at Cloudflare but in 2012 was a delta compression-based service called Railgun. We installed software both on the customer's web server and on our end and thus were able to automatically manage shared dictionaries (in this case version of pages sent over Railgun were used as dictionaries automatically). You definitely get incredible compression results. https://blog.cloudflare.com/c…

I might be naive but isn’t that what rsync is doing?

No. What Railgun did is it enabled the two sides of the connection to agree on a shared dictionary (the most recent version of the page being transmitted) and use that to compress the new page. It required both sides to keep a cache of page versions to compare against.
Post reply on HN