Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We could save petabytes of cache storage with Zstandard and Pingora

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: We could save petabytes of cache storage with Zstandard and Pingora

#11

Post by genxy »

pkulak wrote: post #7
CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.

Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#12

Post by mort96 »

genxy wrote: post #4
MayeulC wrote: post #3

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients. Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor. One of the nice things about…

It has mostly become my go-to as well. Just wish it wasn't a Facebook product.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#13

Post by ncruces »

genxy wrote: post #11
pkulak wrote: post #7

Earlier quoted context omitted.

I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.

Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.

There's nothing about random access at that link.

There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...

Re: We could save petabytes of cache storage with Zstandard and Pingora

#14

Post by articulatepang »

MayeulC wrote: post #3

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

I agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#15

Post by kccqzy »

pkulak wrote: post #7
CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.

Actually zstd internally splits data into frames, and frames can indicate the decompressed data size. So if we control the compressor we can make it so that all frames have the size information; it isn’t exactly seekable but at least it will not need to decompress the resource. https://python-zstandard.readthedocs.io/en/latest/concepts.h...

Given how fast zstd can decompress, this may or may not actually be a win: the time spent waiting for I/O might be so large that the decompression can fit within the wait time.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#16

Post by gopalv »

CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

> I don't see how that's possible if the cache no longer stores the uncompressed data.

Zstd has a seekable format for frames, similar to pigz --independent works.

[1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

Re: We could save petabytes of cache storage with Zstandard and Pingora

#17

Post by nijave »

CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

Idk but btrfs and zfs manage to pull it off

Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces

Re: We could save petabytes of cache storage with Zstandard and Pingora

#18

Post by nijave »

Curious how this application scheme compares to filesystem + transport compression. You'd end up potentially compressing and decompressing more often but the higher software doesn't need to know what's happening and the compression happens in kernel space.

ie btrfs

You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted

Re: We could save petabytes of cache storage with Zstandard and Pingora

#19

Post by nijave »

MayeulC wrote: post #3

> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.

That part is a little bit confusing.

I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source).

Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size)

"Ignore cold assets" makes more sense with that framing

Although if that's the case, the CPU statement still is a bit confusing.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#20

Post by r3trohack3r »

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files.

For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.

There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!

Post reply on HN