Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

121–130 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#121

Post by RivieraKid »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Yes, this is a real problem, verbatim titles are often far from the "optimal" title. In some cases the original title provides almost no information about the content.

The question is what's better than a strict "no editorialization" rule.

Re: Now using Zstandard instead of xz for package compression

#122

Post by gravitas »

Foxboron wrote: post #118
gravitas wrote: post #115

AUR users -- the default settings in /etc/makepkpg.conf (delivered by the pacman package as of 5.2.1-1) are still at xz, you must manually edit your local config: PKGEXT='.pkg.tar.zst' The largest package I always wait on perfect for this scenario is `google-cloud-sdk` (the re-compression is a killer -- `zoom` is another one in AUR that's a beast) so I used it as a test on my laptop here in "real world conditions" (b…

It should be noted that the makepkg.conf file distributed with pacman does not contain the same compression settings as the one used to build official packages. pacman: COMPRESSZST=(zstd -c -z -q -) https://git.archlinux.org/svntogit/packages.git/tree/trunk/m... devtools: COMPRESSZST=(zstd -c -T0 --ultra -20 -) https://github.com/archlinux/devtools/blob/master/makepkg-x8...

The man page for zstd mentions that using the --ultra flag will cause decompression to take more RAM as well when used to compress. Does this indicate a huge increase in memory to decompress, or just a trivial amount per package, say something large like... `libreoffice-fresh`? Or `go`? They're two of the largest main repo packages I have installed... (followed by linux-firmware)

Re: Now using Zstandard instead of xz for package compression

#123

Post by nullc »

Annatar wrote: post #75

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

Per the post, the speedup on decompress is _13x_ while the size is 1.008x.

For those figures, this will be better total time for you if your computer network connection is faster than about 1.25mbit/sec. For a slow arm computer with an XZ decompress speed of 3MB/s the bandwidth threshold for a speedup drops to _dialup_ speeds.

And no matter how slow your network connection is and how fast your computer is you'll never take more than 0.8% longer with this change.

For many realistic setups it will be faster, in some cases quite a bit. Your 54MB XZ host should be about 3% faster if you're on a 6mbit/sec link-- assuming your disk can keep up. A slow host that decompresses xz at 3MB/s w/ a 6mbit link would a wopping 40% faster.

Re: Now using Zstandard instead of xz for package compression

#125

Post by JeremyNT »

I learned about this one the hard way when I went to update a really crufty (~ 1 year since last update) Arch system I use infrequently the other day. I had failed to update my libarchive version prior to the change and the package manager could not process the new format.

Luckily updating libarchive manually with an intermediate version resolved my issue and everything proceeded fine.

This is a good change, but it's a reminder to pay attention to the Arch Linux news feed, because every now and then something important will change. The maintainers provided ample warning about this change there (and indeed I had updated by other systems in response) so we procrastinators really had no excuse :)

Re: Now using Zstandard instead of xz for package compression

#126

Post by telendram »

gravitas wrote: post #122
Foxboron wrote: post #118

Earlier quoted context omitted.

It should be noted that the makepkg.conf file distributed with pacman does not contain the same compression settings as the one used to build official packages. pacman: COMPRESSZST=(zstd -c -z -q -) https://git.archlinux.org/svntogit/packages.git/tree/trunk/m... devtools: COMPRESSZST=(zstd -c -T0 --ultra -20 -) https://github.com/archlinux/devtools/blob/master/makepkg-x8...

The man page for zstd mentions that using the --ultra flag will cause decompression to take more RAM as well when used to compress. Does this indicate a huge increase in memory to decompress, or just a trivial amount per package, say something large like... `libreoffice-fresh`? Or `go`? They're two of the largest main repo packages I have installed... (followed by linux-firmware)

Without `--ultra`, the decompression memory budget is capped at 8 MB. At `--ultra -20`, it's increased to 32 MB.

That's still less than XZ, which reaches 64 MB.

Re: Now using Zstandard instead of xz for package compression

#127

Post by Macha »

yjftsjthsd-h wrote: post #24
viraptor wrote: post #21

Earlier quoted context omitted.

tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

It's actually (at least in the implicit tar -xf foo.tar.gz) made it to all major implementations by now. OpenBSD was the sole exception last time I checked, and OpenBSD tar only untars, it won't decompress even with flags.

Re: Now using Zstandard instead of xz for package compression

#128

Post by zerogara »

Most of the results published show very little positive or negative speed in decompression, where is all this -1300% coming from?

edit: Sorry, my fault that was decompression RAM I was thinking about, not speed, although I was influenced by my test that without measuring both xz and zstd seemed instant.

Re: Now using Zstandard instead of xz for package compression

#129

Post by Dylan16807 »

rat9988 wrote: post #106
nmstoker wrote: post #93

Earlier quoted context omitted.

Saying it's "yet another victim" seems slightly too emotive to me. If people can't read the source site's domain after the headline then I agree there wouldn't be much context, but equally, if they can't read that, surely their best solution is to adjust the zoom level in the browser. It's clear you won't get complete context from the headline list plus domain, but a hint of it is provided and if you want more you cl…

Even using the source domain isn't informative enough. The alternative headline is better. You are being too charitable to an inferior title.

"archlinux.org" is less informative than "Arch Linux"?

I'm sympathetic to disliking the change, but that's taking it to an extreme.

Re: Now using Zstandard instead of xz for package compression

#130

Post by zerogara »

SamWhited wrote: post #88
beanaroo wrote: post #22

Earlier quoted context omitted.

This isn't a function of an AUR helper but rather makepkg itself. In makepkg.conf, ommit compression by specifying: PKGEXT='.pkg.tar' More information can be found at https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks

Then it happens for packages I build myself (and want compressed) and those that I just want to install.

If you care about space more than you care about speed you may want to stick with xz, it is hard to beat or impossible by zstd. So set your own priorities rather than adopt the ones of Arch devs.

As long as there will be support within the tools for xz individual builders of packages for their own use can use either or more.

Post reply on HN