Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression efficiency with shared dictionaries in Chrome

developer.chrome.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Compression efficiency with shared dictionaries in Chrome

#51

Post by jauntywundrkind »

IshKebab wrote: post #47

Ah damn I thought this was going to be available to JavaScript. Would be amazing for one use case I have (an HTML page containing inline logs from a load of commands, many of which are substantially similar).

That would be an excellent web standard!!

There's wasm modules that do similar but having it bakes into the browser could allow for further optimization than what's possible with wasm. https://github.com/bokuweb/zstd-wasm

I have no idea if it's possible but I wonder if a webgpu port could be made? Alternatively, for your use case, maybe you could try applying something like Basis Universal; a fast compression system for textures, that it seems there are some webgpu loaders for... Maybe that could be bent to encoding/deciding text?

Re: Compression efficiency with shared dictionaries in Chrome

#52

Post by jgrahamc »

Scaevolus wrote: post #41
jgrahamc wrote: post #10

Earlier quoted context omitted.

I don't think your characterization of why Railgun was deprecated is accurate. From the blog post you link to: “I use Railgun for performance improvements.” Cloudflare has invested significantly in performance upgrades in the eight years since the last release of Railgun. This list is not comprehensive, but highlights some areas where performance can be significantly improved by adopting newer services relative to us…

Right, but isn't that part of the general trend of bandwidth becoming far cheaper in the last decade along with dynamic HTML becoming a smaller fraction of total transit? A 95%+ reduction in bandwidth usage for dynamic server-side-rendered HTML is much less important in 2023 than 2013.

I think it's related to the size of the Cloudflare network and how good its connectivity is (and our own fibre backbone). But on the eyeball side bandwidth isn't the only game in town: latency is the silent killer.

Re: Compression efficiency with shared dictionaries in Chrome

#53

Post by raggi »

What I really want: dictionaries derived from the standards and standard libraries (perhaps once a year or somesuch), which I'd use independently of build system gunk, and while it wouldn't be the tightest squeeze you can get, it would make my non-built assets get very close to built asset size for small to medium sized deployments.

Re: Compression efficiency with shared dictionaries in Chrome

#54

Post by pyrolistical »

hinkley wrote: post #44
saagarjha wrote: post #17

Even putting aside CORS because I don’t even want to think about how this plays well with requests to another (tracking?) domain, this still doesn’t seem worth it. The explicit use case seems to be that it basically tells the server when you last visited the site based on which dictionary you have and then it gives you the moral equivalent of a delta update. Except, most browsers are working hard to expire data of th…

I think I’d feel better with a fixed set of dictionaries based on a corpus that gets updated every year to match new patterns of traffic and specifications. Even if it’s less efficient.

Ya. Where is accept-encoding: zstandard-d-es2024

Where it encodes js files with a known dictionary that is ideal for es2024

Re: Compression efficiency with shared dictionaries in Chrome

#55

Post by Wingy »

hinkley wrote: post #46
saagarjha wrote: post #18

Earlier quoted context omitted.

So Brotli already contains a dictionary that is trained on web traffic. I think the thing here is that Google wants to make sending YouTube 1.1 more efficient if you already have YouTube 1.0, but they can’t put YouTube 1.0 into the browser.

This is something game devs have been doing for decades. If you want to delta 1.0 to 1.1 that’s server side work you do once at deployment or build time, not on every request.

What happens when you release 1.2 and someone who has 1.0 visits? Do you generate a delta for every past version at build time?

Re: Compression efficiency with shared dictionaries in Chrome

#56

Post by hinkley »

Wingy wrote: post #55
hinkley wrote: post #46

Earlier quoted context omitted.

This is something game devs have been doing for decades. If you want to delta 1.0 to 1.1 that’s server side work you do once at deployment or build time, not on every request.

What happens when you release 1.2 and someone who has 1.0 visits? Do you generate a delta for every past version at build time?

You mean if a user who hasn’t visited the site in a year comes back?

They download 1.2 because 1.0 is no longer in their browser cache, that’s what.

The web is easier than games because “files at rest” are much more volatile on the web.

Re: Compression efficiency with shared dictionaries in Chrome

#57

Post by hinkley »

pyrolistical wrote: post #54
hinkley wrote: post #44

Earlier quoted context omitted.

I think I’d feel better with a fixed set of dictionaries based on a corpus that gets updated every year to match new patterns of traffic and specifications. Even if it’s less efficient.

Ya. Where is accept-encoding: zstandard-d-es2024 Where it encodes js files with a known dictionary that is ideal for es2024

And here’s one tuned for react, and one for svelte…

Re: Compression efficiency with shared dictionaries in Chrome

#58

Post by lynguist »

jgrahamc wrote: post #4

The very first project I worked on at Cloudflare but in 2012 was a delta compression-based service called Railgun. We installed software both on the customer's web server and on our end and thus were able to automatically manage shared dictionaries (in this case version of pages sent over Railgun were used as dictionaries automatically). You definitely get incredible compression results. https://blog.cloudflare.com/c…

I might be naive but isn’t that what rsync is doing?
Post reply on HN