Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

41–50 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#41

Post by ComputerGuru »

shmerl wrote: post #39
ComputerGuru wrote: post #38

Earlier quoted context omitted.

We are talking about decompression speed and not encryption. Decompression is necessarily deterministic.

May be the point is that compressed package can change every time, which is an issue for reproducible builds idea many distros now are using. Though I'm not sure why parallelized xz can't behave in predictable fashion.

No, I mean you don’t need to parallel compress. The compression speeds don’t matter, and are compatible with single- or multi-threaded decompression.

Re: Now using Zstandard instead of xz for package compression

#42

Post by hinkley »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Archlinux: Now using Zstandard instead of xz for package...

should be allowed. But I'm not sure that it is.

Re: Now using Zstandard instead of xz for package compression

#43

Post by Lammy »

Dylan16807 wrote: post #36
Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Most of the text on the front page is that size and that color of gray. If it's not easy to read, then the problem is between the css and your screen. Not the title rules.

I'm talking about the difference in size and contrast between the actual headline and the trailing HN sitebit "(archlinux.org)". The final size they end up on my screen is irrelevant to my point, because my point is about the size and contrast difference _between_ the two, whatever final sizes those might happen to be. The default HN stylesheet calls for 10pt and 8pt for those, respectively, so it's not like I'm just making this up. I'm saying the verbatim title rule is a poor fit here because it took a relevant (central, even!) part of the headline and moved it to a spot of secondary importance and size. There are cases where I defend the rule, but right now I am talking about this case and only this case :)

Re: Now using Zstandard instead of xz for package compression

#44

Post by Carpetsmoker »

_ZeD_ wrote: post #37
yjftsjthsd-h wrote: post #24

Earlier quoted context omitted.

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

With GNU tar, the -a flag is not needed

You don't need it with libarchive-based BSD tar either. It can also extract zipfiles, ISO archives, and many other formats with just "tar xf file.zip".

Re: Now using Zstandard instead of xz for package compression

#45

Post by maxpert »

I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.

Re: Now using Zstandard instead of xz for package compression

#46

Post by JonathonW »

Lammy wrote: post #40
JonathonW wrote: post #31

Earlier quoted context omitted.

BSD tar on my Mac (running 10.15.2) has -a for tarfile creation (-c mode); it always autodetects the compression format in extraction mode (-z, -j, etc. are ignored if -x is specified). Not sure when either behavior would've been introduced; the somewhat-older machine I tested on (running 10.13) does not have -a but does have the autodetection behavior on extract. -I, on the other hand (which, in gnutar, specifies a…

As a FreeBSD user I was curious since I wouldn't want to learn to depend on an option only to find it doesn't exist in version n-1, and today I learned that FreeBSD's tar is contributed code from libarchive and not an in-tree thing like I assumed! https://github.com/freebsd/freebsd/blob/master/usr.bin/tar/M... Looks like that option got merged in 2013: https://github.com/freebsd/freebsd/blame/24ac2fb6ec0610e60ac... T…

This got me digging a little further, and (as a consequence) I just learned something new and fun: the current libarchive-based BSD tar can create and extract a ton [1] of other formats, including ZIP, RAR, and 7Zip files!

That'll save me some time next time I need to extract one at the command line; I always end up having to look at unzip's man page to make sure it's actually going to do what I want, and the 7zip command line utility's kind of funky (and not installed by default most places). But 'tar -xvf filename' is permanently burned into my mind and pretty much always does exactly what I want.

[1] https://www.freebsd.org/cgi/man.cgi?query=libarchive-formats...

Re: Now using Zstandard instead of xz for package compression

#47

Post by kbumsik »

> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.

Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.

Re: Now using Zstandard instead of xz for package compression

#48

Post by shmerl »

ComputerGuru wrote: post #41
shmerl wrote: post #39

Earlier quoted context omitted.

May be the point is that compressed package can change every time, which is an issue for reproducible builds idea many distros now are using. Though I'm not sure why parallelized xz can't behave in predictable fashion.

No, I mean you don’t need to parallel compress. The compression speeds don’t matter, and are compatible with single- or multi-threaded decompression.

Compression speed can matter in general (to improve build times).

For xz, you need to compress with chunking (and may be indexing for more benefit), in order to allow parallel decompression to begin with. Otherwise xz produces a blob which you can't split into independent parts during decompression, which makes using many decompression threads pointless.

But yes, if parallel compression is creating non determinism, you can do all the compression work with chunking without parallelism, still allowing parallel decompression. But I'm not sure why it even has to create non determinism in the first place.

Re: Now using Zstandard instead of xz for package compression

#49

Post by Dylan16807 »

Lammy wrote: post #43
Dylan16807 wrote: post #36

Earlier quoted context omitted.

Most of the text on the front page is that size and that color of gray. If it's not easy to read, then the problem is between the css and your screen. Not the title rules.

I'm talking about the difference in size and contrast between the actual headline and the trailing HN sitebit "(archlinux.org)". The final size they end up on my screen is irrelevant to my point, because my point is about the size and contrast difference _between_ the two, whatever final sizes those might happen to be. The default HN stylesheet calls for 10pt and 8pt for those, respectively, so it's not like I'm just…

So even if the url got bumped to 12pt, you'd complain if the rest was 15? I think that's weird.

As long as it's on the same line as the title and easily legible, I really don't see a problem.

And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.

Re: Now using Zstandard instead of xz for package compression

#50

Post by cmurf »

filereaper wrote: post #17

Apparently this is how to use Zstd with tar if anyone else was wondering: tar -I zstd -xvf archive.tar.zst https://stackoverflow.com/questions/45355277/how-can-i-decom... Hopefully there's another option added to tar that simplifies this if this compression becomes mainstream.

Or

    tar -acf blah.tar.zst blah/
-a figures it out from the filename extension, and its zst not zstd.
Post reply on HN