Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

21–30 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#21

Post by viraptor »

filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

tar accepts `-a` for format autodetection for a while now. You can do:

    tar -axf archive.tar.whatever
and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

Re: Now using Zstandard instead of xz for package compression

#22

Post by beanaroo »

SamWhited wrote: post #15
beatgammit wrote: post #14

Earlier quoted context omitted.

I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.

I love the AUR, but every single time I have to wait for it to compress Firefox nightly, and then wait for it to immediately decompress it again because the only reason I was building the package in the first place was to install it I about lose my mind. Hopefully this helps, but I really wish AUR helpers would just disable compression and call it a day so I don't have to go mess with config files that would also cha…

This isn't a function of an AUR helper but rather makepkg itself.

In makepkg.conf, ommit compression by specifying:

    PKGEXT='.pkg.tar'
More information can be found at https://wiki.archlinux.org/index.php/Makepkg#Tips_and_tricks

Re: Now using Zstandard instead of xz for package compression

#24

Post by yjftsjthsd-h »

viraptor wrote: post #21
filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

tar accepts `-a` for format autodetection for a while now. You can do: tar -axf archive.tar.whatever and it should work for gz, bz2, Z, zstd, and probably more. (verified works for zstd on gnu tar 1.32)

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

Re: Now using Zstandard instead of xz for package compression

#25

Post by yjftsjthsd-h »

beatgammit wrote: post #14
ncmncm wrote: post #2

Zstd has an enormous advantage in compression and, especially, decompression speed. It often doesn't compress quite as much, but we don't care as much as we once did. We rebuild packages more than we once did. This looks like a very good move. Debian should follow suit.

I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.

Why did you re-enable it? Seems to work fine in my experience?

Re: Now using Zstandard instead of xz for package compression

#27

Post by ben0x539 »

filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

Use aunpack imo: https://linux.die.net/man/1/aunpack

Re: Now using Zstandard instead of xz for package compression

#28

Post by shmerl »

esotericn wrote: post #23
shmerl wrote: post #16

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

Multithreaded xz is non-deterministic and so it's not a candidate.

How is it non deterministic? Works pretty consistently for me with pixz.

Re: Now using Zstandard instead of xz for package compression

#29

Post by netfl0 »

jpgvm wrote: post #5
yjftsjthsd-h wrote: post #4

> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades ( https://wiki.archlinux.org/index.php/System_maintenance#Par…

Arch is actually surprisingly stable and even with infrequent updates on the order of months still upgrades cleanly most of the time. The caveats to this were the great period of instability when switching to systemd, changing the /usr/lib layout, etc but those changes are now pretty far in the past.

I remember that time. I had successfully migrate 2 systems to systemd from init. One was a production server. I felt like a genius at the time. Of course all the arch devs did all the real work :)

(I wanted the challenge of running arch in production just to learn, good times)

Re: Now using Zstandard instead of xz for package compression

#30

Post by chungy »

filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

tar automatically detects and supports unpacking zstd-compressed archives (as well as other compression types). there's no reason to use -x combined with other compression flags.

For compression, you can use "-c -I zstd"

Post reply on HN