Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smaller and faster data compression with Zstandard

code.facebook.com

Post reply on HN
Print view View original

131–140 of 166 posts

Re: Smaller and faster data compression with Zstandard

#131

Post by gcp »

ac29 wrote: post #103
tessela wrote: post #87

Earlier quoted context omitted.

except for the PATENTS file =/

Curious what your issue is with it -- it basically says "if you dont sue us, we wont sue you". Thats about as good as I can expect from a large tech company these days with regards to patents.

Compare it to the Opus patent license (also including a retaliation clause), which includes grants from Broadcom, Mozilla and Microsoft.

Using zstd gives Facebook a free license on ALL your patents.

Using Opus gives Facebook only a license on patents that apply to Opus.

So no, large tech companies can and have given MUCH better grants for compression tech, than Facebook is doing.

Re: Smaller and faster data compression with Zstandard

#132

Post by eln1 »

tmd83 wrote: post #118
binarycrusader wrote: post #116

Earlier quoted context omitted.

> I think he is the first one to write a practical fast arithmetic coder using ANS. I don't think he is the first; although RAD game tools has been cagey about the details, many strongly suspect their recently announced Kraken, etc. use ANS in some form and some of their previous products; see the discussion here: https://news.ycombinator.com/item?id=11583898 ...and here: http://encode.ru/threads/2492-Kraken-compress…

Yeah read about Kraken a while ago and found the encode.ru thread you pointed. Its unfortunate that we can't compare that side by side. Those guys seem another of those compression geniuses. Overall the world of compression is moving very fast and well these days. But even with that Yann might be the first. Yan's work on FSE is old [0] and if I'm reading this correctly his work on FSE is a mix of his own work and Jar…

Before Yann, this ANS variant was implemented by Andrew Polar in 2008: http://www.ezcodesample.com/abs/abs_article.html Here is a list of implementations: http://encode.ru/threads/2078-List-of-Asymmetric-Numeral-Sys...

Re: Smaller and faster data compression with Zstandard

#134

Post by socmag »

ctur wrote: post #26

Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and the video will be posted. He can answer the most technical questions about Zstandard, but I may be able to answer some as well; we both work on compression at Facebook.

Hey quick question, and sounds awesome.

If I wanted to use this in the pipeline of my servers Journaling system, is there any requirement that I restart the stream periodically.

That is to say, should I use it per journal entry (probably not a good idea for short messages), for the entire uptime of the writer, or with periodic restarts?

Obviously can measure and find out for myself, but wondered if you had any thoughts. Thanks!

Re: Smaller and faster data compression with Zstandard

#135

Post by nimrody »

morecoffee wrote: post #115

A recent compression discussion I saw involved how do compressors fare on uncompressible input? For example, suppose you wanted to add compression to all your outbound network traffic. What would happen if there was mixed compressible traffic along with the uncomressible kind? A common case would be sending HTML along with JPEG. Good compressors can't squeeze any more out of a JPEG, but they can back off fast and go…

Dropbox say they can losslessly compress JPEG files (~20% saving):

https://blogs.dropbox.com/tech/2016/07/lepton-image-compress...

Re: Smaller and faster data compression with Zstandard

#136

Post by gcp »

nimrody wrote: post #135
morecoffee wrote: post #115

A recent compression discussion I saw involved how do compressors fare on uncompressible input? For example, suppose you wanted to add compression to all your outbound network traffic. What would happen if there was mixed compressible traffic along with the uncomressible kind? A common case would be sending HTML along with JPEG. Good compressors can't squeeze any more out of a JPEG, but they can back off fast and go…

Dropbox say they can losslessly compress JPEG files (~20% saving): https://blogs.dropbox.com/tech/2016/07/lepton-image-compress...

Known techniques. JPEG uses Huffman coding for entropy coding, you can replace that with arithmetic coding. This requires knowing the format details, though.

Re: Smaller and faster data compression with Zstandard

#137

Post by mkesper »

rwmj wrote: post #81
AceJohnny2 wrote: post #36

The modern trend of compressors is to use more memory to achieve speed. This is good if you're using big-iron cloud computers... "Zstandard has no inherent limit and can address terabytes of memory (although it rarely does). For example, the lower of the 22 levels use 1 MB or less. For compatibility with a broad range of receiving systems, where memory may be limited, it is recommended to limit memory usage to 8 MB.…

Minix 1.5 for 8086 had a slightly mad decompress program that would fork itself up to 4 times in order to address enough memory to decompress certain *.Z files: https://github.com/jbruchon/elks/blob/master/elkscmd/minix1/...

Didn't know there were still people working on elks...

Re: Smaller and faster data compression with Zstandard

#138

Post by gliptic »

caf wrote: post #109
AceJohnny2 wrote: post #36

The modern trend of compressors is to use more memory to achieve speed. This is good if you're using big-iron cloud computers... "Zstandard has no inherent limit and can address terabytes of memory (although it rarely does). For example, the lower of the 22 levels use 1 MB or less. For compatibility with a broad range of receiving systems, where memory may be limited, it is recommended to limit memory usage to 8 MB.…

I don't think 8MB is the smallest preset, the text you quoted says that the lower levels use "1 MB or less". The concern I have is that this makes it sound like the compressor can choose how much memory the decompressor will need to use. Does this mean that zstd can't be used in a potentially adversarial environment? (Eg. is there a denial-of-service vector here by forcing the server to use large amounts of memory to…

It will not use (much) more memory than the size of the output in any case. 8MB is the window here, which just means the decompressor can discard data that falls outside this 8MB window as it is decompressing.

Re: Smaller and faster data compression with Zstandard

#139

Post by Cyan4973 »

morecoffee wrote: post #115

A recent compression discussion I saw involved how do compressors fare on uncompressible input? For example, suppose you wanted to add compression to all your outbound network traffic. What would happen if there was mixed compressible traffic along with the uncomressible kind? A common case would be sending HTML along with JPEG. Good compressors can't squeeze any more out of a JPEG, but they can back off fast and go…

Zstd pass faster over incompressible data. Expect something > 1 GB/s
Post reply on HN