Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

141–150 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#141

Post by JyrkiAlakuijala »

imtringued wrote: post #120
dictum wrote: post #107

Earlier quoted context omitted.

Brotli has wide browser support ( https://caniuse.com/#feat=brotli ) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-sit…

That's interesting. Brotli has wide browser support although its less than 5 years old but webp is reaching a decade and Safari still doesn't support it...

WebP has an excellent lossless image compressor (like PNG just 25-40 % more dense), but the lossy format has weaknesses that people focused on, and slowed down the adoption. The initial lossy encoder had weaknesses in quality -- it had bugs or was a port of a video coder. Nowadays, the quality is much better, but the format forces YUV420 coding (does not allow YUV444 coding) which limits the quality of colors and fine textures.

Re: Now using Zstandard instead of xz for package compression

#142

Post by snvzz »

Annatar wrote: post #132
snvzz wrote: post #113

Earlier quoted context omitted.

The extra decompression complexity might be a joke on a Zen2 server, but it definitely is not in older systems. If this was netbsd m68k, you'd probably easily understand.

I use xz on my A1200 all of the time, and Amiga is the stereotypical system where maximum possible compression matters over everything else. Don't make assumptions about me.

I applaud your patience. Even with my vampire, I'll use something faster whether at all possible.

May I ask, why xz over, say, PAQ8PF?

Re: Now using Zstandard instead of xz for package compression

#143

Post by JyrkiAlakuijala »

telendram wrote: post #126
gravitas wrote: post #122

Earlier quoted context omitted.

The man page for zstd mentions that using the --ultra flag will cause decompression to take more RAM as well when used to compress. Does this indicate a huge increase in memory to decompress, or just a trivial amount per package, say something large like... `libreoffice-fresh`? Or `go`? They're two of the largest main repo packages I have installed... (followed by linux-firmware)

Without `--ultra`, the decompression memory budget is capped at 8 MB. At `--ultra -20`, it's increased to 32 MB. That's still less than XZ, which reaches 64 MB.

The respective flag for brotli would be `--large_window 25 --quality 11`

Brotli defines memory use as log2 on command line, i.e., 32 MB = 1 zstd uses a lookup table where the number given by the user is mapped to a decoding-time memory use. The user just needs to look it up if they want to control decoder memory use.

If one benchmarks zstd with `20` and brotli with `20`, zstd may be using 32 MB of decoding memory, where one is specifying 1 MB for brotli. By default zstd tends to use 8 MB for decoding (however it is variable with encoding effort setting) and brotli 4 MB for decoding (not changing with the encoding effort setting).

Re: Now using Zstandard instead of xz for package compression

#144

Post by kdeldycke »

viraptor wrote: post #21
filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

Thanks for that tip, I was able to simplify the magic extract() function from my .bash_profile that is relying on file extension to figure the format of the archive: https://github.com/kdeldycke/dotfiles/commit/8120778f3b968a6...

Re: Now using Zstandard instead of xz for package compression

#145

Post by ncmncm »

bufferoverflow wrote: post #136
WinonaRyder wrote: post #83

Zstandard is awesome! Earlier last year I was doing some research that involved repeatedly grepping through over a terabyte of data, most of which were tiny text files that I had to un-zip/7zip/rar/tar and it was painful (maybe I needed a better laptop). With Zstd I was able to re-compress the whole thing down to a few hundred gigs and use ripgrep which solved the problem beautifully. Out of curiosity I tested compre…

Here's a compression benchmark. http://pages.di.unipi.it/farruggia/dcb/ Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin. LZ4 is a ahead of Snappy in the decompression speed.

I find these numbers for Snappy entirely implausible.

The numbers I know about are wrong: zstd always beats gzip for compression ratio.

I will need to do my own testing.

Re: Now using Zstandard instead of xz for package compression

#146

Post by tatersolid »

imtringued wrote: post #120
dictum wrote: post #107

Earlier quoted context omitted.

Brotli has wide browser support ( https://caniuse.com/#feat=brotli ) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-sit…

That's interesting. Brotli has wide browser support although its less than 5 years old but webp is reaching a decade and Safari still doesn't support it...

> but webp is reaching a decade and Safari still doesn't support it...

That’s a philosophical objection. For a long while Mozilla also was of the opinion that WebP is not “better enough” than JPEG/PNG to warrant the addition of another image format which the entire web must support forever using only one available implementation.

Plus I think there are still some unresolved patent claims on the VP8/9 video codec (which are the basis for WebP).

Re: Now using Zstandard instead of xz for package compression

#147

Post by rat9988 »

Dylan16807 wrote: post #129
rat9988 wrote: post #106

Earlier quoted context omitted.

Even using the source domain isn't informative enough. The alternative headline is better. You are being too charitable to an inferior title.

"archlinux.org" is less informative than "Arch Linux"? I'm sympathetic to disliking the change, but that's taking it to an extreme.

With this title, I thought it was a post advocating the usage of Zstandard or vanting its technical merits, and that it was posted on archlinux.org.

When you take out information, don't expect people to have the correct guess.

Re: Now using Zstandard instead of xz for package compression

#148

Post by ncmncm »

ncmncm wrote: post #145
bufferoverflow wrote: post #136

Earlier quoted context omitted.

Here's a compression benchmark. http://pages.di.unipi.it/farruggia/dcb/ Looks like Snappy beats both LZ4 and Zstd in compression speed and compression ratio, by a huge margin. LZ4 is a ahead of Snappy in the decompression speed.

I find these numbers for Snappy entirely implausible. The numbers I know about are wrong: zstd always beats gzip for compression ratio. I will need to do my own testing.

I have tested snzip 1.0.4.

It compresses about as well as lz4, but more slowly. It also decompresses more slowly.

It is faster than zstd -1, but compresses less well.

It is possible that it does better with certain kinds of data, but 12x remains implausible.

Apparently the current file format has suffix ".sz".

Re: Now using Zstandard instead of xz for package compression

#149

Post by SamWhited »

zerogara wrote: post #130
SamWhited wrote: post #88

Earlier quoted context omitted.

Then it happens for packages I build myself (and want compressed) and those that I just want to install.

If you care about space more than you care about speed you may want to stick with xz, it is hard to beat or impossible by zstd. So set your own priorities rather than adopt the ones of Arch devs. As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.

I don't really care about space all that much, but packages I build tend to get uploaded and the people downloading them may not have fast internet. Meanwhile, packages built by my AUR helper I care about speed (seriously, it takes ages to compress then immediately decompress firefox). The problem isn't that I want to optimize for one or the other, it's that AUR helpers generally have a different need than I do when building my packages myself, but for some reason AUR helpers don't override the compression setting for just their install. Probably due to caching like I said which means they can't assume everyone will want compression off all the time, but I'm not sure, that's just a guess.

Re: Now using Zstandard instead of xz for package compression

#150

Post by bscphil »

Lammy wrote: post #135
RivieraKid wrote: post #121

Earlier quoted context omitted.

Yes, this is a real problem, verbatim titles are often far from the "optimal" title. In some cases the original title provides almost no information about the content. The question is what's better than a strict "no editorialization" rule.

The exact guideline is "If the title includes the name of the site, please take it out, because the site name will be displayed after the link", and I think that wording speaks from an outdated mindset where every submission is a standalone web page that _has_ a title, for one thing. This submission is a web page with its own title, of course, but that makes it sound like the guideline hasn't been rethought in too-lo…

> The exact guideline is "If the title includes the name of the site, please take it out, because the site name will be displayed after the link", and I think that wording speaks from an outdated mindset where every submission is a standalone web page that _has_ a title, for one thing.

I suspect the original intent of the rule was to get rid of pointless redundancy in the title. "The 10 craziest things you don't know about X - clickbait.com" is the sort of thing you see very often in the element, but it adds no new information. Actually, you'll notice even Hacker News posts have " | Hacker News" appended to them.

In an article about Arch Linux, the text "Arch Linux" is much less likely to be redundant than an article about something else that just happens to be on the archlinux.org domain.

Post reply on HN