Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zstandard v1.5.0

github.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Zstandard v1.5.0

#21

Post by ac29 »

rektide wrote: post #3

When can we bring this to the web? Zstd aka RFC8478[1] is so good. That it can continue to improve at all feels almost unbelievable, but @Cyan4973 &al continue to make it faster, somehow. Especially on mobile, with large assets, I feel like zstd's lightning fast decompression time could be a huge win. It used to be that Brotli was the obvious choice for achieving high compression, but it doesn't feel so clear to me n…

Caddy supports Zstd encoding: https://caddyserver.com/docs/caddyfile/directives/encode

On the client end, curl does: https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html

Re: Zstandard v1.5.0

#23

Post by duskwuff »

meltedcapacitor wrote: post #11
rektide wrote: post #3

When can we bring this to the web? Zstd aka RFC8478[1] is so good. That it can continue to improve at all feels almost unbelievable, but @Cyan4973 &al continue to make it faster, somehow. Especially on mobile, with large assets, I feel like zstd's lightning fast decompression time could be a huge win. It used to be that Brotli was the obvious choice for achieving high compression, but it doesn't feel so clear to me n…

Brotli is such an ugly hack (hardcoded dictionary with a snapshot of the world as it looked from Mountain View on some random day...), the quicker it dies the better.

The contents of that hardcoded dictionary are really weird, too. It includes a lot of bizarrely specific entries like:

    in popular culture
    Holy Roman Emperor
    It is important to
    examples include the
    have speculated that
    aria-hidden="true">·
Many of the English phrases (like "in popular culture"!) are highly characteristic of content from the English Wikipedia. I've speculated before that this may be the result of the use of a compression benchmark which included (or consisted entirely of) content from that site, such as https://cs.fit.edu/~mmahoney/compression/textdata.html

If you're curious, here's a full dump of the dictionary:

https://gist.github.com/duskwuff/8a75e1b5e5a06d768336c8c7c37...

Re: Zstandard v1.5.0

#24

Post by viraptor »

infogulch wrote: post #14
aidenn0 wrote: post #8

Zstd is so much better than the commonly-used alternatives that I get mildly annoyed when given a .tar.{gz,xz,bz2} it's not like it's a huge deal, but a much smaller file (compared to gz) or similarly sized with much faster decompression (comared to xz, bz2) just makes me a tiny bit happier.

Your comment made me curious what a Zstandard-compressed tar file's extension would be, and apparently it's .tar.zst

I understand they probably tried to keep to 3 letters (because history), but I'm unreasonably annoyed they didn't go with .zstd :(

Even the mime type is full application/zstd

Re: Zstandard v1.5.0

#25

Post by adeno4 »

apendleton wrote: post #17
aidenn0 wrote: post #8

Zstd is so much better than the commonly-used alternatives that I get mildly annoyed when given a .tar.{gz,xz,bz2} it's not like it's a huge deal, but a much smaller file (compared to gz) or similarly sized with much faster decompression (comared to xz, bz2) just makes me a tiny bit happier.

I agree with the general premise that there's no reason to ever use gzip anymore (unless you're in an environment where you can't install stuff), but interestingly my experience with the tradeoffs is apparently not the same as yours. I tend to find that zstd and gzip give pretty similar compression ratios for the things I tend to work with, but that zstd is way faster, and that xz offers better compression ratios tha…

> there's no reason to ever use gzip anymore

There is: convenience. There are many examples, such as Wireshark support reading *.pcap.gz directly.

Re: Zstandard v1.5.0

#26

Post by chrismorgan »

rektide wrote: post #3

When can we bring this to the web? Zstd aka RFC8478[1] is so good. That it can continue to improve at all feels almost unbelievable, but @Cyan4973 &al continue to make it faster, somehow. Especially on mobile, with large assets, I feel like zstd's lightning fast decompression time could be a huge win. It used to be that Brotli was the obvious choice for achieving high compression, but it doesn't feel so clear to me n…

You cite RFC 8478 as though it being an RFC gives it weight, but always remember when looking at IETF RFCs to check the Category, which is Informational in this case, not Standards Track.

> Despite use of the word "standard" as part of its name, readers are advised that this document is not an Internet Standards Track specification; it is being published for informational purposes only.

This is an IETF-stream document, which I believe (though I’m not certain) implies that it’s at least a little more than just Facebook saying “here, can we publish this?”, but it hasn’t come through any working group, so it hasn’t been subject to the full IETF experience which is so helpful at improving things by collaboration.

(It’s also obsoleted by RFC 8878, to which the same caveats apply.)

Re: Zstandard v1.5.0

#27

Post by hifly »

duskwuff wrote: post #23
meltedcapacitor wrote: post #11

Earlier quoted context omitted.

Brotli is such an ugly hack (hardcoded dictionary with a snapshot of the world as it looked from Mountain View on some random day...), the quicker it dies the better.

The contents of that hardcoded dictionary are really weird, too. It includes a lot of bizarrely specific entries like: in popular culture Holy Roman Emperor It is important to examples include the have speculated that aria-hidden="true">· Many of the English phrases (like "in popular culture"!) are highly characteristic of content from the English Wikipedia. I've speculated before that this may be the result of the u…

however the idea seems sound? I can't help wondering why Google who can fund GP-3, couldn't come up with a better initial dictionary? especially for small responses it is a big win

Re: Zstandard v1.5.0

#28

Post by SergeAx »

Fun fact: when we started compressing our analytical events (furry JSON arrays with lots of context) we've dropped about 60% of our total inbound traffic. I was afraid that compression will eat client's battery, but we eventually got a reverse effect: saving energy on radio part, being WiFi or 4G/5G.

Re: Zstandard v1.5.0

#29

Post by nijave »

SergeAx wrote: post #28

Fun fact: when we started compressing our analytical events (furry JSON arrays with lots of context) we've dropped about 60% of our total inbound traffic. I was afraid that compression will eat client's battery, but we eventually got a reverse effect: saving energy on radio part, being WiFi or 4G/5G.

For background transfers reducing processing time can also maximize sleep/low power mode, too (Doze is one of them on Android but I think there's some other low power states)

There's some interesting (imo) research about optimizing cpu governors on Android that details some of the tradeoffs (like running high power on a single core might be better in some cases than longer at low power if the core can be hotplugged/disabled sooner)

Re: Zstandard v1.5.0

#30

Post by rubyist5eva »

terrelln wrote: post #5
rubyist5eva wrote: post #4

Is this the same "zstd" compression used in Fedora's btrfs transparent block level compression? I have been thoroughly impressed with it in Fedora 34. If that's true, I had no idea that it was a Facebook project. Color me shocked.

Yeah, it is. The Linux kernel is currently using zstd-1.3.1, and I'm working on getting it updated to the latest zstd version.

Thank you for the great work!
Post reply on HN