Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

11–20 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#11

Post by cjbillington »

yjftsjthsd-h wrote: post #4

> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades ( https://wiki.archlinux.org/index.php/System_maintenance#Par…

pacman-static existed already, and can be used to fix some of the most broken systems in a variety of circumstances. So, they didn't make it just for this, might as well mention it as the right tool to fix the problem should it occur.

Re: Now using Zstandard instead of xz for package compression

#14

Post by beatgammit »

ncmncm wrote: post #2

Zstd has an enormous advantage in compression and, especially, decompression speed. It often doesn't compress quite as much, but we don't care as much as we once did. We rebuild packages more than we once did. This looks like a very good move. Debian should follow suit.

I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.

Re: Now using Zstandard instead of xz for package compression

#15

Post by SamWhited »

beatgammit wrote: post #14
ncmncm wrote: post #2

Zstd has an enormous advantage in compression and, especially, decompression speed. It often doesn't compress quite as much, but we don't care as much as we once did. We rebuild packages more than we once did. This looks like a very good move. Debian should follow suit.

I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.

I love the AUR, but every single time I have to wait for it to compress Firefox nightly, and then wait for it to immediately decompress it again because the only reason I was building the package in the first place was to install it I about lose my mind. Hopefully this helps, but I really wish AUR helpers would just disable compression and call it a day so I don't have to go mess with config files that would also change my manual package building routine.

EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression.

Re: Now using Zstandard instead of xz for package compression

#17

Post by filereaper »

Apparently this is how to use Zstd with tar if anyone else was wondering:

  tar -I zstd -xvf archive.tar.zst
https://stackoverflow.com/questions/45355277/how-can-i-decom...

Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

Re: Now using Zstandard instead of xz for package compression

#20

Post by shmerl »

rubicks wrote: post #18
shmerl wrote: post #16

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

I give thanks every day for pxz. I can churn out apt indices so much faster relative to the alternative.

For general purposes, I like using pixz which is indexable in comparison: https://github.com/vasi/pixz

Do you know if Debian is using parallelized XZ or not with apt / dpkg?

Post reply on HN