Now using Zstandard instead of xz for package compression • Hacker News Board

Re: Now using Zstandard instead of xz for package compression

#71

Post by ncmncm » Sun, Jan 05, 2020, 9:25 AM UTC

integricho wrote: post #65

ncmncm wrote: post #55

Earlier quoted context omitted.

Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.

After reading these comments, I can't help but wonder, what is the benefit of Zstd over lz4? Why didn't they switch to lz4 if it was the speed of the algorithm that they favored even with marginally worse compression ratios?

Where Zstd will reduce, say, 3x, Lz4 reduces only 2.5x. This doesn't seem very different until you look at it from the other end: my .zst file is 3.3 GB, but the .lz4 would have been 4 GB, which is 700 MB more.

Was a time when 700 MB mattered; it was as much as you could get onto a CD.

So, there is a place for each. I would set up the process to use Lz4 when testing, and Zstd for actual delivery to download archives.

In some circumstances, particularly when using a shared file server, Lz4 can be quite a lot faster than writing and reading data uncompressed.

Re: Now using Zstandard instead of xz for package compression

#72

Post by telendram » Sun, Jan 05, 2020, 9:28 AM UTC

shmerl wrote: post #16

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

A little known fact is that parallel XZ do compress worse than XZ ! I measured pixz as being approximately ~2% worse than xz. That's because input is split into independent chunks.

In comparison, the 0.8% of zstd looks like a bargain.

Re: Now using Zstandard instead of xz for package compression

#73

Post by shmerl » Sun, Jan 05, 2020, 9:30 AM UTC

telendram wrote: post #72

shmerl wrote: post #16

Was XZ used in parallelized fashion? Otherwise comparing is kind of pointless. Single threaded XZ decompression is way too slow.

A little known fact is that parallel XZ do compress worse than XZ ! I measured pixz as being approximately ~2% worse than xz. That's because input is split into independent chunks. In comparison, the 0.8% of zstd looks like a bargain.

Is 0.8% with maximum compression? It's surprising the difference is so small.

Re: Now using Zstandard instead of xz for package compression

#74

Post by rurban » Sun, Jan 05, 2020, 9:31 AM UTC

loeg wrote: post #52

I'd love to see Zstandard accepted in other places where the current option is only the venerable zlib. E.g., git packing, ssh -C. It's got more breadth and is better (ratio / cpu) than zlib at every point in the curve where zlib even participates.

Also zlib is horrible code. Went away with disgust after finding critical errors (that time only patched in master).

Re: Now using Zstandard instead of xz for package compression

#75

Post by Annatar » Sun, Jan 05, 2020, 9:52 AM UTC

I couldn't care less about decompression speed, because the bottleneck is the network, which means that I want my packages as small as possible. Smaller packages mean faster installation; at 54 MB/s or faster decompression rate of xz, I couldn't care less about a few milliseconds saved during decompression. For me, this decision is dumbass stupid.

Re: Now using Zstandard instead of xz for package compression

#76

Post by telendram » Sun, Jan 05, 2020, 9:52 AM UTC

shmerl wrote: post #73

telendram wrote: post #72

Earlier quoted context omitted.

A little known fact is that parallel XZ do compress worse than XZ ! I measured pixz as being approximately ~2% worse than xz. That's because input is split into independent chunks. In comparison, the 0.8% of zstd looks like a bargain.

Is 0.8% with maximum compression? It's surprising the difference is so small.

0.8% is with Arch's default settings. It's fairly strong, but not the strongest, to preserve cpu during compression.

zstd is used at level 20, but it can compress more. Levels can go up to 22, and (complex) advanced commands are available to compress data even more.

Re: Now using Zstandard instead of xz for package compression

#77

Post by esotericn » Sun, Jan 05, 2020, 9:59 AM UTC

ComputerGuru wrote: post #38

esotericn wrote: post #23

Earlier quoted context omitted.

Multithreaded xz is non-deterministic and so it's not a candidate.

We are talking about decompression speed and not encryption. Decompression is necessarily deterministic.

The compression speed is also an issue for developers. In many cases the compression step takes longer than the rest of the build.

Re: Now using Zstandard instead of xz for package compression

#78

Post by nwah1 » Sun, Jan 05, 2020, 10:00 AM UTC

Squithrilve wrote: post #59

nwah1 wrote: post #19

I wonder if they will switch to using zstd for mkinititcpio

mkinitcpio is being replaced with dracut so zstd won't probably happen.

Man page says zstd is an option on dracut

http://man7.org/linux/man-pages/man5/dracut.conf.5.html

Re: Now using Zstandard instead of xz for package compression

#79

Post by nwah1 » Sun, Jan 05, 2020, 10:01 AM UTC

yjftsjthsd-h wrote: post #26

nwah1 wrote: post #19

I wonder if they will switch to using zstd for mkinititcpio

I thought that was user configurable? Or do you mean by default?

Yea, by default. Last time I tried it manually, the kernel wouldn't boot. Best to have these things handled for you.

Re: Now using Zstandard instead of xz for package compression

#80

Post by Foxboron » Sun, Jan 05, 2020, 10:19 AM UTC

Squithrilve wrote: post #59

nwah1 wrote: post #19

I wonder if they will switch to using zstd for mkinititcpio

mkinitcpio is being replaced with dracut so zstd won't probably happen.

Well, that is a bit on a 50/50 coinflip currently. There has been intentions but we need some collaboration from upstream to make this happen.

It's not set in stone currently.