Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Now using Zstandard instead of xz for package compression

archlinux.org

Post reply on HN
Print view View original

51–60 of 155 posts

Re: Now using Zstandard instead of xz for package compression

#51

Post by Lammy »

Dylan16807 wrote: post #49
Lammy wrote: post #43

Earlier quoted context omitted.

I'm talking about the difference in size and contrast between the actual headline and the trailing HN sitebit "(archlinux.org)". The final size they end up on my screen is irrelevant to my point, because my point is about the size and contrast difference _between_ the two, whatever final sizes those might happen to be. The default HN stylesheet calls for 10pt and 8pt for those, respectively, so it's not like I'm just…

So even if the url got bumped to 12pt, you'd complain if the rest was 15? I think that's weird. As long as it's on the same line as the title and easily legible, I really don't see a problem. And it's not a spot of secondary importance. If it was still in the title, making it longer, the spot where you see the url would have title in it.

I'm glad it's not a problem for you. It is for me, though, and the point of accessibility is to enable use by everyone.

Re: Now using Zstandard instead of xz for package compression

#53

Post by loeg »

maxpert wrote: post #45

I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.

Zstd has "fast" negative levels (-5, -4, ... -1, 1, ..., 22). -4 or -5 are purportedly comparable (but not quite as good) as LZ4.

Re: Now using Zstandard instead of xz for package compression

#54

Post by ncmncm »

loeg wrote: post #52

I'd love to see Zstandard accepted in other places where the current option is only the venerable zlib. E.g., git packing, ssh -C. It's got more breadth and is better (ratio / cpu) than zlib at every point in the curve where zlib even participates.

Wireshark! Wireshark!

Also lz4, of course.

Re: Now using Zstandard instead of xz for package compression

#55

Post by ncmncm »

kbumsik wrote: post #47

> Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup. Impressive. As a AUR package maintainer I am also wondering how the compression speed is though.

Compression speed is many, many, many times faster than xz, and (only) much faster than gzip. Really, only lz4 beats it.

Re: Now using Zstandard instead of xz for package compression

#56

Post by ncmncm »

maxpert wrote: post #45

I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.

Use Lz4 where latency matters, Zstd if you can afford some CPU.

I have a server that spools off the entire New York stock and options market every day, plus Chicago futures, using Lz4. But when we copy to archive, we recompress it with Zstd, in parallel using all the cores that were tied up all day.

There is not much size benefit to more than compression level 3: I would never use more than 6. And, there's not much CPU benefit for less than 1, even though it will go into negative numbers; switch to Lz4 instead.

Re: Now using Zstandard instead of xz for package compression

#57

Post by ncmncm »

loeg wrote: post #53
maxpert wrote: post #45

I’ve used LZ4 and Snappy in production for compressing cache/mq payloads. This is on a service serving billions of clicks in a day. So far very happy with the results, I know zstd requires more CPU than LZ4 or snappy on average but has someone used it under heavy traffic loads on web services. I am really interested trying it out but at the same time held back by “don’t fix it if it ain’t broken”.

Zstd has "fast" negative levels (-5, -4, ... -1, 1, ..., 22). -4 or -5 are purportedly comparable (but not quite as good) as LZ4.

Better to just use Lz4, then.

Re: Now using Zstandard instead of xz for package compression

#58

Post by ncmncm »

golergka wrote: post #13

I used zstd for on-the-fly compression of game data for p2p multiplayer synchronization, and got 2-5x as much data (depends on the payload type) in each TCP packet. Sad that it still doesn't get much adoption in the industry.

Zstd knows how to use a user-supplied dictionary at each end. I hope you are doing that.

But if latency matters you might better use lz4.

Re: Now using Zstandard instead of xz for package compression

#60

Post by xiaq »

yjftsjthsd-h wrote: post #4

> If you nevertheless haven't updated libarchive since 2018, all hope is not lost! Binary builds of pacman-static are available from Eli Schwartz' personal repository, signed with their Trusted User keys, with which you can perform the update. I am a little shocked that they bothered; Arch is rolling release and explicitly does not support partial upgrades ( https://wiki.archlinux.org/index.php/System_maintenance#Par…

I guess that's why it is provided by an individual, instead of as an officially supported solution.
Post reply on HN