111–120 of 155 posts
Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…
Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…
I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.
If this was netbsd m68k, you'd probably easily understand.
Earlier quoted context omitted.
Sure, and I've done partial upgrades and it was mostly fine:) It just surprised me to see the devs going out of their way to support it on volunteer time. On the other hand, maybe that's exactly the reason; maybe someone said "hey look, I can make static packages that are immune to library changes! I guess I'll publish these in case they're useful". Open source is fun like that:)
Also, Arch devs probably run Arch servers, and I'd not be surprised if some of those have uptimes in hundreds of days.
https://git.archlinux.org/infrastructure.git/
Now, official infra doesn't reach hundreds of days. But personal systems might :)
PKGEXT='.pkg.tar.zst'
After warming up all my caches with a few pre-builds to try and keep it fair by reducing disk I/O, here's a sampling of the results:
xz defaults - Size: 33649964
real 2m23.016s
user 1m49.340s
sys 0m35.132s
----
zst defaults - Size: 47521947
real 1m5.904s
user 0m30.971s
sys 0m34.021s
----
zst mpthread - Size: 47521114
real 1m3.943s
user 0m30.905s
sys 0m33.355s
I did not seem to notice any appreciable difference in adding the `--threads=0` to `COMPRESSZST=` (from the Arch wiki), they both consistently gave me right around what you see above. This was compression only testing which is where my wait time is when upgrading these packages, huge improvement with zst seen here...
Earlier quoted context omitted.
Zstd has "fast" negative levels (-5, -4, ... -1, 1, ..., 22). -4 or -5 are purportedly comparable (but not quite as good) as LZ4.
Better to just use Lz4, then.
If you know your use case well and are sure the situation won't change (or don't mind swapping compression algorithms when they do), then lz4 still has a solid niche, especially where compression speed matters more than decompression speed. But in many if not most cases I'd say it's probably a kind of premature optimization at this point, even if you think you're close to lz4's sweet spot.
> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.
"... decompression time dropped to 14% of what it was..." (s/14/actual_value)
AUR users -- the default settings in /etc/makepkpg.conf (delivered by the pacman package as of 5.2.1-1) are still at xz, you must manually edit your local config: PKGEXT='.pkg.tar.zst' The largest package I always wait on perfect for this scenario is `google-cloud-sdk` (the re-compression is a killer -- `zoom` is another one in AUR that's a beast) so I used it as a test on my laptop here in "real world conditions" (b…
pacman:
COMPRESSZST=(zstd -c -z -q -)
devtools:
COMPRESSZST=(zstd -c -T0 --ultra -20 -)
Also one more benefit of zstd compression, that is not widely noted - a zstd file conpressed with multiple threads is binary the same as file compressed with single thread. So you can use multi threaded compression and you will end up with the same file cheksum, which is very important for package signing.
On the other hand xz, which has been used before, produces a binary different file if compressed by single or multiple threads. This basucally precludes multi threaded compression at package build time, as the compressed file checksums would not match if the package was rebuild with a different number of compression threads. (the unpacked payload will be always the same, but the compressed xz file will be binary different)
Earlier quoted context omitted.
It would be great to see better compression supported by browsers.
Brotli has wide browser support ( https://caniuse.com/#feat=brotli ) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-sit…