Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller and faster data compression with Zstandard

code.facebook.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Smaller and faster data compression with Zstandard

#61

Post by wongarsu »

AceJohnny2 wrote: post #36

The modern trend of compressors is to use more memory to achieve speed. This is good if you're using big-iron cloud computers... "Zstandard has no inherent limit and can address terabytes of memory (although it rarely does). For example, the lower of the 22 levels use 1 MB or less. For compatibility with a broad range of receiving systems, where memory may be limited, it is recommended to limit memory usage to 8 MB.…

Any modern server that handles a thousand or more concurrent connections on commodity hardware already uses only about as many threads as there are processor cores. In that architecture it's trivial to also limit the number of compression threads to the number of processor cores. That architecture gives the best performance and very low memory use.

In the mid-2000 it was still accepted norm to spawn one thread for each connection, where memory usage of the compressor would have been a problem. I doubt that it's a problem with today's software architecture.

Re: Smaller and faster data compression with Zstandard

#63

Post by Grishnakh »

rspeer wrote: post #16
Grishnakh wrote: post #11

Looks very interesting, however I'm not impressed by the name. "Zstandard"??? With ".zstd" as the extension? I don't like it. They should have named it letter -zip, along the lines of gzip, bzip, and xzip, with the extension letter z. "fz" would have been a good one since they work at Facebook.

...yeah, why didn't they claim part of a namespace that only has room for 26 (or 36) things? Everyone else is doing it!

I don't see the problem: only 3 members of that namespace are currently claimed (4 out of the 36-member namespace: 7z), so we have room for 23 (or 32) more compression standards before running out. We've been using gzip for, what, 20 years now? Only recently have we gotten xz. At this rate, we won't run out of compression standards using this scheme for roughly 153 years. And after that, we could always start using capital Zs, like the old "compress" standard that used the .Z extension. Or we could go to a 3-letter extension ending in z, such as ".fbz", which gives you 676 more options, and 4507 years. Considering that general-purpose data compression really hasn't moved that much since the DEFLATE algorithm took over, and only recently had any real change with the advent of LZMA (used in p7zip and xz), and perhaps this new zstd (too soon to tell), my time estimates here are probably too short.

Re: Smaller and faster data compression with Zstandard

#64

Post by Grishnakh »

VanillaCafe wrote: post #14
Grishnakh wrote: post #11

Looks very interesting, however I'm not impressed by the name. "Zstandard"??? With ".zstd" as the extension? I don't like it. They should have named it letter -zip, along the lines of gzip, bzip, and xzip, with the extension letter z. "fz" would have been a good one since they work at Facebook.

https://en.wiktionary.org/wiki/bikeshedding

This isn't bikeshedding. Bikeshedding is about quibbling over unimportant details. Names are critically and absolutely important. Lots of great things have been hobbled or ruined by poorly-chosen names. A terrible name can cause something worthy to be ignored in favor of something inferior but with a better name. And you don't need to even be competent in the inner workings of a project to criticize its name or suggest better ones. The people who name cars aren't the same people who design the engine-control algorithms for them.

If you disagree, what do you think of naming your kid "Adolph Hitler [lastname]"? Most people agree that a name like that will cause great harm to a child growing up because of the constant ridicule and ostracization he'd inevitably face. That's an extreme example, but names are important.

Note that I don't think this project's name is horrible, but I don't think it's very good either, and could be a lot better.

Re: Smaller and faster data compression with Zstandard

#65

Post by esaym »

>> "It is written in highly portable C, making it suitable for practically every platform used today"

I love C, it is not the enemy everyone makes it out to be.

It's already in debian: https://packages.debian.org/stretch/zstd and judging by the small requirements,it is portable indeed.

Re: Smaller and faster data compression with Zstandard

#66

Post by ohitsdom »

wongarsu wrote: post #57
ohitsdom wrote: post #50

I'm a complete dunce when it comes to compression and how it fits in the industry, so help me out here. Say that everyone accepts that Zstandard is amazing and we should start using it. What would the adoption process look like? I understand individual programs could implement it since they would handle both compression and decompression, but what about the web? Would HTTP servers first have to add support, then brow…

>what about the web The browser sends the server a request header indicating which compression methods it understands. Current Firefox for example sends Accept-Encoding: gzip, deflate, br meaning the server is free to send a response compressed with either gzip, deflate or brotli. Or the server can choose to send the data uncompressed. This means the adoption path for the web would be an implemtation in at least one…

Nice, so really browsers and servers can implement it independently. Seems elegant, I like that browsers can advertise multiple compression formats. Thanks for the info!

Re: Smaller and faster data compression with Zstandard

#69

Post by espadrine »

The following link points to a fairly good benchmark / tool that showcases the tradeoffs in real life: since (de)compression takes time, what is the fastest way to transmit data at a given transfer speed?

https://quixdb.github.io/squash-benchmark/unstable/#transfer...

Spoilers: zstd wins at ethernet and wifi (and is among the best in 4G), lz4 wins at hard drive encryption… both were designed by the same author.

Post reply on HN