Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

151–155 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#151
post #37

Earlier quoted context omitted.

I think that's a GNU extension, so obviously fine on Arch, but probably not on ex. MacOS (Darwin) or Alpine (busybox) by default.

With GNU tar, the -a flag is not needed

Note that it doesn't work with compression.

  tar cf example.tar.zst example
creates an uncompressed tar archive.

  tar caf example.tar.zst example
creates a zstd compressed archive.

Re: Now using Zstandard instead of xz for package compression

#152
post #107

Earlier quoted context omitted.

Brotli has wide browser support ( https://caniuse.com/#feat=brotli ) and comes closer to zstd in compression ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at least Cloudflare seems to serve it by default (EDIT: must be enabled per-sit…

Brotli compresses about 5-10 % more than zstd. Benchmarks showing equal compression performance use different window sizes (smaller window sizes for brotli) or do not run at maximum compression density. https://github.com/google/brotli/issues/642 is the best 3rd party documentation of this behavior. zstd does decompress fast, but this is not free. The cost is the compression density -- and lesser streaming properties…

The linked series of comments (which, to be clear, I've only skimmed — there's a ton there) show zstd 22 sometimes coming behind Brotli 11d29, sometimes ahead on compression ratio; usually coming ahead of Brotli 11 on compression ratio; ~5x faster on compression throughput and ~2-2.5x faster on decompress throughput. To cherry-pick some numbers (the table after "259,707,904 bytes long open_watcom_1.9.0-src.tar", dated "TurboBench: - Mon Apr 30 07:51:32 2018"):

  Name         | Comp. size | Comp. ratio | C. MB/s | D. MB/s
  brotli 11d29 | 21117099   | 8.1         | 0.52    | 515
  zstd 22      | 22249772   | 8.6         | 2.32    | 1270
  brotli 11    | 22898691   | 8.8         | 0.57    | 662
So in that particular instance, zstd 22 comes out about 5% worse (+1.1 MB over Brotli 11d29's 20.1 MiB) on compressed size, but 3% better (-640kiB) vs Brotli 11 at 21.8 MiB. So... maximum compression is within a small margin; compression and decompression speeds are much quicker.

I think it's fair to say that zstd struggles the most at the extremes. On the fast extreme it loses (marginally) to lz4; on the slow extreme it (maybe) loses (marginally) to brotli. But it's relatively quick across the spectrum and provides a lot of flexibility.

It may make sense to continue to use Brotli or xz for static assets that are compressed infrequently and read often. But for something like HTTP Content-Encoding, where dynamic pages are compressed on the fly? Zstd would shine here, over both Brotli and (vanilla) zlib. (I know Chrome has some hacked up zlib on the client side, but I do not know too much about it.)

Re: Now using Zstandard instead of xz for package compression

#153
post #142

Earlier quoted context omitted.

I use xz on my A1200 all of the time, and Amiga is the stereotypical system where maximum possible compression matters over everything else. Don't make assumptions about me.

I applaud your patience. Even with my vampire, I'll use something faster whether at all possible. May I ask, why xz over, say, PAQ8PF?

Because my Amiga: volume is actually an NFS share on my Solaris 10 central storage server in the basement. With xz(1) I'm easily portable across systems.

And you're talking to a guy who had his C128D spend all nights crunching intros + pic + game at 1 MHz in Cruel Cruncher 2.5+ and Time Cruncher 5.0 before that. xz(1) on a MC68030 or the Vampire is super fast in comparison.

Re: Now using Zstandard instead of xz for package compression

#154
post #142

Earlier quoted context omitted.

I applaud your patience. Even with my vampire, I'll use something faster whether at all possible. May I ask, why xz over, say, PAQ8PF?

Because my Amiga: volume is actually an NFS share on my Solaris 10 central storage server in the basement. With xz(1) I'm easily portable across systems. And you're talking to a guy who had his C128D spend all nights crunching intros + pic + game at 1 MHz in Cruel Cruncher 2.5+ and Time Cruncher 5.0 before that. xz(1) on a MC68030 or the Vampire is super fast in comparison.

So Zstd won't run/build on Solaris or AmigaOS?

Solaris 10 as in the last one from Sun (Damn Oracle...) is an interesting choice. Does openindiana not run well on the old hardware?

Re: Now using Zstandard instead of xz for package compression

#155
post #154

Earlier quoted context omitted.

Because my Amiga: volume is actually an NFS share on my Solaris 10 central storage server in the basement. With xz(1) I'm easily portable across systems. And you're talking to a guy who had his C128D spend all nights crunching intros + pic + game at 1 MHz in Cruel Cruncher 2.5+ and Time Cruncher 5.0 before that. xz(1) on a MC68030 or the Vampire is super fast in comparison.

So Zstd won't run/build on Solaris or AmigaOS? Solaris 10 as in the last one from Sun (Damn Oracle...) is an interesting choice. Does openindiana not run well on the old hardware?

I don't know whether it'll build because I'm interested in maximum compression -- time spent compressing is immaterial to me since it's only done once. It's like compiling -- I don't care how long it takes if it generates fast machine code, because it's running the generated machine code that will matter many, many times afterwards.

Is Core2 Quad with 8GB of RAM or a Sun X4100 M2 "older hardware"?

Post reply on HN