Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller and faster data compression with Zstandard

code.facebook.com

Post reply on HN
Print view View original

101–110 of 166 posts

Re: Smaller and faster data compression with Zstandard

#101

Post by CyberShadow »

ktta wrote: post #17
CyberShadow wrote: post #7

Earlier quoted context omitted.

> the link attached says https://github.com/Cyan4973/zstd but then redirects to https://github.com/facebook/zstd . Anyone know why? I believe this is just GitHub's standard behavior when a repository is moved to another namespace. We recently renamed https://github.com/D-Programming-Language to https://github.com/dlang , and e.g. https://github.com/D-Programming-Language/dmd redirects in the same way.

Oh, D language is dlang now? Looks like you guys did a redesign of the website and stuff too! I hear good things about D, keep up the great work!

Thanks! To clarify, we didn't rename the language, just the GitHub organization. However, dlang.org is what we've been using as the canonical address for the website, and "dlang" is a much more searchable term than just "D". (Now, if only Google would stop auto-correcting "dlang" to "golang"...)

Re: Smaller and faster data compression with Zstandard

#102

Post by twic »

Grishnakh wrote: post #63
rspeer wrote: post #16

Earlier quoted context omitted.

...yeah, why didn't they claim part of a namespace that only has room for 26 (or 36) things? Everyone else is doing it!

I don't see the problem: only 3 members of that namespace are currently claimed (4 out of the 36-member namespace: 7z), so we have room for 23 (or 32) more compression standards before running out. We've been using gzip for, what, 20 years now? Only recently have we gotten xz. At this rate, we won't run out of compression standards using this scheme for roughly 153 years. And after that, we could always start using c…

Perhaps we should require the less common algorithms to have a longer prefix.

Re: Smaller and faster data compression with Zstandard

#103

Post by ac29 »

tessela wrote: post #87
mrmrben wrote: post #49

Really nice work compared to what I consider to be the quite bad Brotli -- an incredibly slow compression standard that only ended up in browsers because it was created by Google.

except for the PATENTS file =/

Curious what your issue is with it -- it basically says "if you dont sue us, we wont sue you". Thats about as good as I can expect from a large tech company these days with regards to patents.

Re: Smaller and faster data compression with Zstandard

#104

Post by cooper12 »

hueving wrote: post #99
riboflava wrote: post #92

Earlier quoted context omitted.

It's also possible to implement a decompressor in javascript to support browsers which don't do it natively. The performance would likely suck but if you're truly bandwidth constrained and don't mind users having a bit of a lag, it's an option...

Assuming the size of the decompressor isn't larger than the savings you gained from using this compression algo over another...

Well the decompressor would be cached by the browser so it would pay off for repeat visitors. And if it gets used enough, the user will already have downloaded the decompressor from another site.

Re: Smaller and faster data compression with Zstandard

#105

Post by gribbly »

bryanlarsen wrote: post #31
cbr wrote: post #20

The plot of compression ratio against speed for the various compression levels is pretty helpful for understanding its performance: https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9... "The x-axis is a decreasing logarithmic scale in megabytes per second; the y-axis is the compression ratio achieved." I'd love to see a version of this chart that also included Brotli. (And I'm somewhat surprised Brotli isn'…

I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...

>Our internal testing shows that it performs very poorly for binary 3D vector data.

Uh ? What format of data was this ?

I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared.

It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings:

bro -quality 11 -window 24

zstd --ultra -22

So I find it baffling that you find brotli to be poor for any binary data, could you share the data in question ?

Re: Smaller and faster data compression with Zstandard

#107

Post by haughter »

gribbly wrote: post #105
bryanlarsen wrote: post #31

Earlier quoted context omitted.

I thought that brotli was tuned for typical web workloads, that it contained a dictionary tuned for web workloads. Our internal testing shows that it performs very poorly for binary 3D vector data. So a test between zstd and brotli would show brotli in a poor light if it used a mixed corpus, but a test between zstd and brotli on a web corpus would give an advantage to brotli...

>Our internal testing shows that it performs very poorly for binary 3D vector data. Uh ? What format of data was this ? I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared. It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings: bro -quality 1…

What is the compression time for brotli to achieve compression ratios comparable to LZMA on these types of data sets: https://github.com/google/brotli/issues/165

I understand that brotli is incredibly slow as compared to LZMA when using these high ratio settings (q11, w24), so slow as to be impractical in production even in a write-once, read-many scenario if you have any non-trivial amount of data being regularly produced. I do not want to have a farm of machines just to handle the brotli compression load of our data sets just because it is 10x slower than LZMA.

Re: Smaller and faster data compression with Zstandard

#108

Post by _wmd »

wongarsu wrote: post #61
AceJohnny2 wrote: post #36

The modern trend of compressors is to use more memory to achieve speed. This is good if you're using big-iron cloud computers... "Zstandard has no inherent limit and can address terabytes of memory (although it rarely does). For example, the lower of the 22 levels use 1 MB or less. For compatibility with a broad range of receiving systems, where memory may be limited, it is recommended to limit memory usage to 8 MB.…

Any modern server that handles a thousand or more concurrent connections on commodity hardware already uses only about as many threads as there are processor cores. In that architecture it's trivial to also limit the number of compression threads to the number of processor cores. That architecture gives the best performance and very low memory use. In the mid-2000 it was still accepted norm to spawn one thread for ea…

A server like this could only work by buffering an entire response before compressing it once, requiring (compressed+uncompressed) bytes temporary space. In reality most servers of the design you mention operate streamily, flushing the compressor's output just in time as the backend fills its input buffer. In designs like that (most of them) a compression context per connection is still required

Re: Smaller and faster data compression with Zstandard

#109

Post by caf »

AceJohnny2 wrote: post #36

The modern trend of compressors is to use more memory to achieve speed. This is good if you're using big-iron cloud computers... "Zstandard has no inherent limit and can address terabytes of memory (although it rarely does). For example, the lower of the 22 levels use 1 MB or less. For compatibility with a broad range of receiving systems, where memory may be limited, it is recommended to limit memory usage to 8 MB.…

I don't think 8MB is the smallest preset, the text you quoted says that the lower levels use "1 MB or less".

The concern I have is that this makes it sound like the compressor can choose how much memory the decompressor will need to use. Does this mean that zstd can't be used in a potentially adversarial environment? (Eg. is there a denial-of-service vector here by forcing the server to use large amounts of memory to decompress my requests?)

Re: Smaller and faster data compression with Zstandard

#110

Post by gribbly »

haughter wrote: post #107
gribbly wrote: post #105

Earlier quoted context omitted.

>Our internal testing shows that it performs very poorly for binary 3D vector data. Uh ? What format of data was this ? I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed within ~5% margin of lzma, and this holds true on other binary data I've compared. It also compressed better than zstd (as in compression ratio) on the same data at their highest respective compression settings: bro -quality 1…

What is the compression time for brotli to achieve compression ratios comparable to LZMA on these types of data sets: https://github.com/google/brotli/issues/165 I understand that brotli is incredibly slow as compared to LZMA when using these high ratio settings (q11, w24), so slow as to be impractical in production even in a write-once, read-many scenario if you have any non-trivial amount of data being regularly pr…

>What is the compression time for brotli to achieve compression ratios comparable to LZMA...

Compression time is indeed the achilles heel of Brotli, which is why it's something I would only use for compress once (and preferably decompress very often) scenarios.

Compared to lzma at it's best compression setting for this particular data (-mx9 -m0=LZMA:d512m:fb273:lc8), brotli took 6 minutes and 4 seconds to compress, while the same data took 1 minute and 47 seconds for lzma.

On the other hand, brotli decompressed the same data in 0.6 seconds, while it took lzma 2.2 seconds.

Post reply on HN