Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

91–100 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#91

Post by Dylan16807 »

asutekku wrote: post #89
Dylan16807 wrote: post #62

Earlier quoted context omitted.

You don't get to declare that something is an accessibility problem just because you don't like it, though. You would rate the exact same formatting good or bad based entirely on whether the text next to it is a couple pixels larger. That does not sound like "accessibility". Whether it's hard to read is an accessibility issue. But that's not your complaint.

I have to agree with the previous poster. Sometimes i have absolutely no clue what the article is about because the original title has been edited and the url is not immediately recognizable.

> Sometimes i have absolutely no clue what the article is about because the original title has been edited and the url is not immediately recognizable.

Yes, that happens... but I'm not sure how it applies to this specific case? The URL is the same two words that were removed from the title.

I'm not defending HN's title edits in general.

Re: Now using Zstandard instead of xz for package compression

#93

Post by nmstoker »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

Saying it's "yet another victim" seems slightly too emotive to me.

If people can't read the source site's domain after the headline then I agree there wouldn't be much context, but equally, if they can't read that, surely their best solution is to adjust the zoom level in the browser.

It's clear you won't get complete context from the headline list plus domain, but a hint of it is provided and if you want more you click the link. Maybe I'm being a little uncharitable but I don't see a big problem here.

Re: Now using Zstandard instead of xz for package compression

#94

Post by ncmncm »

rwmj wrote: post #92

I wish zstd supported seeking and partial decompression ( https://github.com/facebook/zstd/issues/395#issuecomment-535... ). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.

Fun fact: Two zstd files appended is a zstd file.

Also, parallel zstd must have some way to split up the work, that you could maybe use too.

Re: Now using Zstandard instead of xz for package compression

#96

Post by rwmj »

ncmncm wrote: post #94
rwmj wrote: post #92

I wish zstd supported seeking and partial decompression ( https://github.com/facebook/zstd/issues/395#issuecomment-535... ). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.

Fun fact: Two zstd files appended is a zstd file. Also, parallel zstd must have some way to split up the work, that you could maybe use too.

I would suggest reading the github issue that I linked to, you'll see why it's not currently possible.

Re: Now using Zstandard instead of xz for package compression

#97

Post by nloomans »

Lammy wrote: post #35

Meta: This post is yet another victim of the HN verbatim title rule despite the verbatim title making little sense as one of many headlines on a news page. How is "Now using Zstandard instead of xz for package compression" followed by the minuscule low-contrast grey "(archlinux.org)" better than "Arch Linux now using Zstandard instead of xz for package compression" like it was when I originally read this a few hours…

I just woke up to this and was surprised the title was eddied as well. I looked up the guidelines and it looks like I violated the "If the title includes the name of the site, please take it out, because the site name will be displayed after the link." guideline.

Re: Now using Zstandard instead of xz for package compression

#98

Post by the8472 »

kbumsik wrote: post #47

> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.

While the speedup is nice pacman still seems to operate sequentially, i.e. download, then decompress one by one. Decompressing while downloading or decompressing in parallel seems like a low-hanging fruit that hasn't been plucked yet that wouldn't have needed any changes to the compressor.

Re: Now using Zstandard instead of xz for package compression

#99

Post by tempay »

shmerl wrote: post #28
esotericn wrote: post #23

Earlier quoted context omitted.

Multithreaded xz is non-deterministic and so it's not a candidate.

How is it non deterministic? Works pretty consistently for me with pixz.

The bytes of the compressed file are non deterministic and depend on the number of cores used, system load and other “random” factors.

Re: Now using Zstandard instead of xz for package compression

#100

Post by ubercow13 »

Annatar wrote: post #75

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

Why do you care so much about the few extra miliseconds wasted downloading, then? (0.8% size increase is ~ 0). Also don't forget that Arch can also be used on machines with very slow CPU but very fast network connections, like many VPSs. I think this will make a tangible difference on mine. This is also a big improvement for package maintainers and anyone building their own packages without bothering to modify the makepkg defaults, eg. most people using an AUR helper.
Post reply on HN