It could save PyPI petabytes per month of bandwidth, too. (But it seems like this is also caused by broken CI systems failing to cache things locally.)
We could save petabytes of cache storage with Zstandard and Pingora
21–30 of 64 posts
Re: We could save petabytes of cache storage with Zstandard and Pingora
#22Why not serving files compressed if the client supports it even though the origin served an uncompressed file?
They already do this, at least for paid accounts. You can even decide what compression model you want them to serve on your behalf.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#23It could save PyPI petabytes per month of bandwidth, too. (But it seems like this is also caused by broken CI systems failing to cache things locally.)
Yeah. It's worth looking at your own code and infrastructure as well. In my previous job there were dozens of opportunities to realise big savings by swapping out gzip for other compression schemes -- usually zstd, occasionally lz4 or bz2 or brotli. Huge assets took less hard drive space, took less time to download, and took less time to decompress. The differences were not small, and resulted in appreciable improvements both in infra cost and dev productivity.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#24I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
Idk but btrfs and zfs manage to pull it off Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces
It uses some form of keyframing. Not entirely free.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#25Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#26> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime.
Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again.
And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#27I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
Idk but btrfs and zfs manage to pull it off Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces
afaik, ZFS will read an entire Record at a time- and that's the same granularity as its compression.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#28I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
they said they didn't change the behavior for range requests. So, it'll still be the basic no compression. (eg, server side decompression)
Re: We could save petabytes of cache storage with Zstandard and Pingora
#29Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#30Earlier quoted context omitted.
I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.
Which would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive. You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking. Or they have an upper size limit for the file size they compress, since large files are rarely comp…
For the cost of a small amount of metadata the offsets of every MiB or so could be stored. I did something like this with pigz as I was implementing multithreaded compressed and encrypted kernel zone suspend and resume for Solaris.