Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

31–40 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#31

Post by JonathonW »

yjftsjthsd-h wrote: post #24
viraptor wrote: post #21

Earlier quoted context omitted.

tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

BSD tar on my Mac (running 10.15.2) has -a for tarfile creation (-c mode); it always autodetects the compression format in extraction mode (-z, -j, etc. are ignored if -x is specified). Not sure when either behavior would've been introduced; the somewhat-older machine I tested on (running 10.13) does not have -a but does have the autodetection behavior on extract.

-I, on the other hand (which, in gnutar, specifies a compression program to run the output through), appears to actually be GNU-specific. BSD tar makes -I synonymous with -T (specifying a file containing the list of filenames to be extracted or archived).

(Please don't use Zstandard if you care about cross-platform compatibility at all, though-- it's fine in controlled environments like an OS package manager, but I don't have it on my Mac, nor do I have it by default on my Ubuntu server (which is still sitting back on 16.04; I should fix that sooner or later).)

Re: Now using Zstandard instead of xz for package compression

#32

Post by computerfriend »

yjftsjthsd-h wrote: post #4

> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades ( https://wiki.archlinux.org/index.php/System_maintenance#Par…

I have a laptop in another country that I see quite infrequently and I'm pretty happy this exists.

Re: Now using Zstandard instead of xz for package compression

#33

Post by crazysim »

agumonkey wrote: post #8
G4E wrote: post #3

For those who want a TLDR : The trade off is 0.8% increase of package size for 1300% increase in decompression speed. Those numbers come from a sample of 542 packages.

thanks, a great change for those with SSDs

or CPUs!

Re: Now using Zstandard instead of xz for package compression

#35

Post by Lammy »

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page.

How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours ago?

Re: Now using Zstandard instead of xz for package compression

#36

Post by Dylan16807 »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Most of the text on the front page is that size and that color of gray.

If it's not easy to read, then the problem is between the css and your screen. Not the title rules.

Re: Now using Zstandard instead of xz for package compression

#37

Post by _ZeD_ »

yjftsjthsd-h wrote: post #24
viraptor wrote: post #21

Earlier quoted context omitted.

tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

With GNU tar, the -a flag is not needed

Re: Now using Zstandard instead of xz for package compression

#38

Post by ComputerGuru »

esotericn wrote: post #23
shmerl wrote: post #16

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

Multithreaded xz is non-deterministic and so it's not a candidate.

We are talking about decompression speed and not encryption. Decompression is necessarily deterministic.

Re: Now using Zstandard instead of xz for package compression

#39

Post by shmerl »

ComputerGuru wrote: post #38
esotericn wrote: post #23

Earlier quoted context omitted.

Multithreaded xz is non-deterministic and so it's not a candidate.

We are talking about decompression speed and not encryption. Decompression is necessarily deterministic.

May be the point is that compressed package can change every time, which is an issue for reproducible builds idea many distros now are using. Though I'm not sure why parallelized xz can't behave in predictable fashion.

Re: Now using Zstandard instead of xz for package compression

#40

Post by Lammy »

JonathonW wrote: post #31
yjftsjthsd-h wrote: post #24

Earlier quoted context omitted.

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

BSD tar on my Mac (running 10.15.2) has -a for tarfile creation (-c mode); it always autodetects the compression format in extraction mode (-z, -j, etc. are ignored if -x is specified). Not sure when either behavior would've been introduced; the somewhat-older machine I tested on (running 10.13) does not have -a but does have the autodetection behavior on extract. -I, on the other hand (which, in gnutar, specifies a…

As a FreeBSD user I was curious since I wouldn't want to learn to depend on an option only to find it doesn't exist in version n-1, and today I learned that FreeBSD's tar is contributed code from libarchive and not an in-tree thing like I assumed! https://github.com/freebsd/freebsd/blob/master/usr.bin/tar/M...

Looks like that option got merged in 2013: https://github.com/freebsd/freebsd/blame/24ac2fb6ec0610e60ac...

The textual libarchive changelog isn't super clear on it, and I didn't feel like digging further, but appears to have been introduced some time between libarchive 3.0.4 in 2012 and libarchive 3.1.2 in 2013: https://github.com/freebsd/freebsd/commit/366f42737cba40ceb2...

Post reply on HN