Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We could save petabytes of cache storage with Zstandard and Pingora

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: We could save petabytes of cache storage with Zstandard and Pingora

#51

Post by vbezhenar »

r3trohack3r wrote: post #20

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…

I did simpler trick with jars (basically zip archives). Java ecosystem loves huge directories full of jars. If one would just use good compressor like 7z over that directory, it won't compress that good. So I extracted every jar into a separate directory and then compressed them all with 7z. The results were very good.

I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win.

The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#52

Post by TacticalCoder »

walrus01 wrote: post #48
nithril wrote: post #37

One thing that bothered me enough to comment on: "transcoding" doesn't seem like the right term in this context. "Encode/decode" is technically correct, but "compress/decompress" would have made the intent much clearer.

Transcoding is generally used within the context of audio or video codecs, to convert a file from one format to another. And usually from one lossy format to another (eg: not raw uncompressed YUV420, YUV422P video or whatever that is stored in a lossless compression format). It's not clear to me why they're using it in this web page.

Yeah, the following sentence in TFA is weird:

> Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already.

It's because they're already lossily compressed that, precisely, they're the typical targets for transcoding.

EDIT: oops, wrote lossless instead of lossy, sorry about that

Re: We could save petabytes of cache storage with Zstandard and Pingora

#53

Post by andersmurphy »

NostraDavid wrote: post #49
chickenbig wrote: post #40

Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck. It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data…

> compression levels -1, 1 or 2 Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...

There's negative levels for faster decoding and compression. level -1 uses half the CPU of level 1 and with a custom window size can be really good for streaming compression. I think the negative levels go down to -7 if I remember correctly?

Re: We could save petabytes of cache storage with Zstandard and Pingora

#54

Post by andersmurphy »

I'm surprised the faster decompression speed of zstd wasn't mentioned. Feels like a big win for cache (assuming decompress on read).

A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#55

Post by im3w1l »

donavanm wrote: post #26
MayeulC wrote: post #3

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime. Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your…

You are making an argument against caching it at all which is clearly not what they want. So the comparison must be uncompressed storage vs compressed storage.

The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits.

So compression makes the most sense for something you want to keep a long time that will be accessed very rarely. And the least sense for something you will drop very soon and will have many people requesting it.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#56

Post by wood_spirit »

andersmurphy wrote: post #54

I'm surprised the faster decompression speed of zstd wasn't mentioned. Feels like a big win for cache (assuming decompress on read). A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.

Wait, what? Don’t browsers support concatenated zstd streams?

My reading - and I care because I’ve been using concatenation tricks in parquet - is that this is required by the standard.

In the distant past I made a jinja2-like template system for server side rendering that precompressed the static bits and injected the short dynamic bits into the output stream as literals with some basic lz match rewriting in the following chunks etc. This kind of trick ought be better with zstd frames.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#57

Post by squirrellous »

chickenbig wrote: post #40

Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck. It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data…

Riegeli is great! Packed full of features. It could be the standard storage format for serialized pb binary if there were more official support.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#58

Post by andersmurphy »

wood_spirit wrote: post #56
andersmurphy wrote: post #54

I'm surprised the faster decompression speed of zstd wasn't mentioned. Feels like a big win for cache (assuming decompress on read). A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.

Wait, what? Don’t browsers support concatenated zstd streams? My reading - and I care because I’ve been using concatenation tricks in parquet - is that this is required by the standard. In the distant past I made a jinja2-like template system for server side rendering that precompressed the static bits and injected the short dynamic bits into the output stream as literals with some basic lz match rewriting in the fol…

Sadly they do not. They stop after the first frame (at least last time I tested it).

They also don't always support window sizes above 8MB which means levels above 19 can fail to decompress (if they use the more than 8MB of window).

Re: We could save petabytes of cache storage with Zstandard and Pingora

#59

Post by walrus01 »

TacticalCoder wrote: post #52
walrus01 wrote: post #48

Earlier quoted context omitted.

Transcoding is generally used within the context of audio or video codecs, to convert a file from one format to another. And usually from one lossy format to another (eg: not raw uncompressed YUV420, YUV422P video or whatever that is stored in a lossless compression format). It's not clear to me why they're using it in this web page.

Yeah, the following sentence in TFA is weird: > Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already. It's because they're already lossily compressed that, precisely, they're the typical targets for transcoding. EDIT: oops, wrote lossless instead of lossy, sorry about that

Additionally modern fonts are normally vector point data, no? So there's plenty of opportunity for shrinking file size by lossless compession. But I don't think that's what they're talking about here?
Post reply on HN