Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

61–70 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#61

Post by thenewnewguy »

SamWhited wrote: post #15
beatgammit wrote: post #14

Earlier quoted context omitted.

I build packages periodically from the AUR, and compression is the longest part of the process much of the time. For a while, I disabled compression on AUR packages because it was becoming enough of a problem for me to look into solutions. If it's annoying for me, I can imagine it's especially problematic for package maintainers. I can only imagine how much CPU time switching the compression tool will save.

I love the AUR, but every single time I have to wait for it to compress Firefox nightly, and then wait for it to immediately decompress it again because the only reason I was building the package in the first place was to install it I about lose my mind. Hopefully this helps, but I really wish AUR helpers would just disable compression and call it a day so I don't have to go mess with config files that would also cha…

> EDIT: nevermind, this doesn't seem to have made this the default for building packages locally, just for ones you download from the official repos. Guess I'll go change that by hand and then still be sad that I can't have it easily disabled entirely for AUR helpers but build my packages with compression.

I believe you can supply it via an environment variable if your AUR helper has the ability to set those for `makepkg`.

Re: Now using Zstandard instead of xz for package compression

#62

Post by Dylan16807 »

Lammy wrote: post #51
Dylan16807 wrote: post #49

Earlier quoted context omitted.

So even if the url got bumped to 12pt, you'd complain if the rest was 15? I think that's weird. As long as it's on the same line as the title and easily legible, I really don't see a problem. And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.

I'm glad it's not a problem for you. It is for me, though, and the point of accessibility is to enable use by everyone.

You don't get to declare that something is an accessibility problem just because you don't like it, though.

You would rate the exact same formatting good or bad based entirely on whether the text next to it is a couple pixels larger. That does not sound like "accessibility".

Whether it's hard to read is an accessibility issue. But that's not your complaint.

Re: Now using Zstandard instead of xz for package compression

#63

Post by throwGuardian »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Guessing such a rule helps in duplicate submissions detection. But, that should be possible from checking the URL.

Unless one uses a link shortener. Are shorteners permitted on HN?

Re: Now using Zstandard instead of xz for package compression

#64

Post by grzm »

throwGuardian wrote: post #63
Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Guessing such a rule helps in duplicate submissions detection. But, that should be possible from checking the URL. Unless one uses a link shortener. Are shorteners permitted on HN?

Nope. Some sources, such as Medium embed viewer-dependent identifiers in the url which can confound de-duping based on URL alone. I don't know if 'dang et al have figured out a way to handle these cases.

Re: Now using Zstandard instead of xz for package compression

#65

Post by integricho »

ncmncm wrote: post #55
kbumsik wrote: post #47

> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.

Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.

After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?

Re: Now using Zstandard instead of xz for package compression

#66

Post by Lammy »

Dylan16807 wrote: post #62
Lammy wrote: post #51

Earlier quoted context omitted.

I'm glad it's not a problem for you. It is for me, though, and the point of accessibility is to enable use by everyone.

You don't get to declare that something is an accessibility problem just because you don't like it, though. You would rate the exact same formatting good or bad based entirely on whether the text next to it is a couple pixels larger. That does not sound like "accessibility". Whether it's hard to read is an accessibility issue. But that's not your complaint.

I'm happy to agree to disagree. Please forgive me if I am (hopefully!) just misreading the tone of these comments, but this exchange has seemed tiringly mean-spirited and argumentative to me. I'm not trying to convince you of some objective fault in HN's design, just sharing my experience to see if anyone else's is similar. My experiences will still be my reality even if the answer to that question is "no" :)

Re: Now using Zstandard instead of xz for package compression

#67

Post by Dylan16807 »

Lammy wrote: post #66
Dylan16807 wrote: post #62

Earlier quoted context omitted.

You don't get to declare that something is an accessibility problem just because you don't like it, though. You would rate the exact same formatting good or bad based entirely on whether the text next to it is a couple pixels larger. That does not sound like "accessibility". Whether it's hard to read is an accessibility issue. But that's not your complaint.

I'm happy to agree to disagree. Please forgive me if I am (hopefully!) just misreading the tone of these comments, but this exchange has seemed tiringly mean-spirited and argumentative to me. I'm not trying to convince you of some objective fault in HN's design, just sharing my experience to see if anyone else's is similar. My experiences will still be my reality even if the answer to that question is "no" :)

I got snarky because you said "I'm glad it's not a problem for you" when I never said the current design wasn't a problem. I read that as unwarrantedly dismissive!

I'm not disagreeing with your experience, I'm just disagreeing with part of the way you want to fix it.

Re: Now using Zstandard instead of xz for package compression

#68

Post by isatty »

integricho wrote: post #65
ncmncm wrote: post #55

Earlier quoted context omitted.

Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.

After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?

Guessing that 0.8x size increase for 1300% speedup was worth the tradeoff but maybe ≥1.5 size increase or more was not (especially considering a 1300%->2000% increase is not going to be user visible for 99% of the packages).

Re: Now using Zstandard instead of xz for package compression

#69

Post by jacobolus »

loeg wrote: post #52

I'd love to see Zstandard accepted in other places where the current option is only the venerable zlib. E.g., git packing, ssh -C. It's got more breadth and is better (ratio / cpu) than zlib at every point in the curve where zlib even participates.

It would be great to see better compression supported by browsers.

Re: Now using Zstandard instead of xz for package compression

#70

Post by TJSomething »

isatty wrote: post #68
integricho wrote: post #65

Earlier quoted context omitted.

After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?

Guessing that 0.8x size increase for 1300% speedup was worth the tradeoff but maybe ≥1.5 size increase or more was not (especially considering a 1300%->2000% increase is not going to be user visible for 99% of the packages).

It's not 0.8 times size increase, it's a 0.008 times size increase, since the unit is percent. The latter seems pretty marginal to me.
Post reply on HN