Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

131–140 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#131

Post by Annatar »

ubercow13 wrote: post #100
Annatar wrote: post #75

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

Why do you care so much about the few extra miliseconds wasted downloading, then? (0.8% size increase is ~ 0). Also don't forget that Arch can also be used on machines with very slow CPU but very fast network connections, like many VPSs. I think this will make a tangible difference on mine. This is also a big improvement for package maintainers and anyone building their own packages without bothering to modify the ma…

Because size does matter.

Re: Now using Zstandard instead of xz for package compression

#132

Post by Annatar »

snvzz wrote: post #113
Annatar wrote: post #75

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

The extra decompression complexity might be a joke on a Zen2 server, but it definitely is not in older systems. If this was netbsd m68k, you'd probably easily understand.

I use xz on my A1200 all of the time, and Amiga is the stereotypical system where maximum possible compression matters over everything else. Don't make assumptions about me.

Re: Now using Zstandard instead of xz for package compression

#133

Post by shmerl »

tempay wrote: post #99
shmerl wrote: post #28

Earlier quoted context omitted.

How is it non deterministic? Works pretty consistently for me with pixz.

The bytes of the compressed file are non deterministic and depend on the number of cores used, system load and other “random” factors.

Can't you set those parameters during compression to something fixed? Should be doable.

Re: Now using Zstandard instead of xz for package compression

#134

Post by powturbo »

Annatar wrote: post #75

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

There are nice plots [1] to see the tranfer+decompression speedup depening on the network bandwidth.

This is for html web compression, but the results are similar for other datasets. For internet transfer more compression is better than more decompression speed.

You can make your own experiments incl. the plots with turbobench [2]

[1] https://sites.google.com/site/powturbo/home/web-compression [2] https://github.com/powturbo/TurboBench

Re: Now using Zstandard instead of xz for package compression

#135

Post by Lammy »

RivieraKid wrote: post #121
Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Yes, this is a real problem, verbatim titles are often far from the "optimal" title. In some cases the original title provides almost no information about the content. The question is what's better than a strict "no editorialization" rule.

The exact guideline is "If the title includes the name of the site, please take it out, because the site name will be displayed after the link", and I think that wording speaks from an outdated mindset where every submission is a standalone web page that _has_ a title, for one thing. This submission is a web page with its own title, of course, but that makes it sound like the guideline hasn't been rethought in too-long of a time.

For a contrived example of how dated the guideline seems, what if somebody submitted a tweet thread criticizing Twitter the company with a headline/sitebit like "Twitter now banning third-party clients. (twitter.com)". Would it have to be renamed to "Now banning third-party clients. (twitter.com)"? That would make it appear to be a more official statement instead of an unsponsored opinion.

I'm picking on Twitter out of recent memory of this submission of mine a couple weeks ago, where the submission title "Tracking down the true origin of a font used in many games and shareware titles" was 100% my own editorializing for lack of title-worthy material in the linked tweet itself: https://news.ycombinator.com/item?id=21667238

Re: Now using Zstandard instead of xz for package compression

#136

Post by bufferoverflow »

WinonaRyder wrote: post #83

Zstandard is awesome! Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop). With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully. Out of curiosity I tested compre…

Here's a compression benchmark.

http://pages.di.unipi.it/farruggia/dcb/

Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin.

LZ4 is a ahead of Snappy in the decompression speed.

Re: Now using Zstandard instead of xz for package compression

#137

Post by esotericn »

shmerl wrote: post #133
tempay wrote: post #99

Earlier quoted context omitted.

The bytes of the compressed file are non deterministic and depend on the number of cores used, system load and other “random” factors.

Can't you set those parameters during compression to something fixed? Should be doable.

Or you can just use zstd.

The xz tool is not deterministic when compressing. The packaging team might change upstream for a few things, but diving into the innards of a compression tool is expecting a bit much.

Re: Now using Zstandard instead of xz for package compression

#138

Post by ncmncm »

zerogara wrote: post #130
SamWhited wrote: post #88

Earlier quoted context omitted.

Then it happens for packages I build myself (and want compressed) and those that I just want to install.

If you care about space more than you care about speed you may want to stick with xz, it is hard to beat or impossible by zstd. So set your own priorities rather than adopt the ones of Arch devs. As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.

I doubt anybody today has legitimately enough reason to care about space to prefer xz over zstd. The only plausible reason is that some customer, or customer's customer, is only equipped to unpack archaic formats.

Re: Now using Zstandard instead of xz for package compression

#139

Post by correct_horse »

bufferoverflow wrote: post #136
WinonaRyder wrote: post #83

Zstandard is awesome! Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop). With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully. Out of curiosity I tested compre…

Here's a compression benchmark. http://pages.di.unipi.it/farruggia/dcb/ Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin. LZ4 is a ahead of Snappy in the decompression speed.

Similar to how code is read more times than it is written, files are decompressed more times than compressed.

I have not researched this opinion much

Re: Now using Zstandard instead of xz for package compression

#140

Post by JyrkiAlakuijala »

dictum wrote: post #107
jacobolus wrote: post #69

Earlier quoted context omitted.

It would be great to see better compression supported by browsers.

Brotli has wide browser support ( https://caniuse.com/#feat=brotli ) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-sit…

Brotli compresses about 5-10 % more than zstd. Benchmarks showing equal compression performance use different window sizes (smaller window sizes for brotli) or do not run at maximum compression density.

https://github.com/google/brotli/issues/642 is the best 3rd party documentation of this behavior.

zstd does decompress fast, but this is not free. The cost is the compression density -- and lesser streaming properties than brotli.

For typical linux package use, one could save 5 % more in density by moving from zstd to large window brotli. The decompression speed for a typical package would be slowed down by 1 ms, but the decompression could happen during the transfer or file I/O if that is an issue.

Post reply on HN