Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We could save petabytes of cache storage with Zstandard and Pingora

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: We could save petabytes of cache storage with Zstandard and Pingora

#31

Post by genxy »

ncruces wrote: post #13
genxy wrote: post #11

Earlier quoted context omitted.

Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.

There's nothing about random access at that link. There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...

Also external dictionaries https://nigeltao.github.io/blog/2022/zstandard-part-7-dictio...

Re: We could save petabytes of cache storage with Zstandard and Pingora

#32

Post by mgerdts »

nijave wrote: post #17
CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

Idk but btrfs and zfs manage to pull it off Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces

ZFS compresses recordsize or volblocksize chunks down to some whole number of disk blocks, as determined by 1 >> ashift. In practice, this typically means that each 128k chunk gets compressed to some number of sequential 512 or 4096 byte blocks. These compressed blocks are referenced by block pointers that contain flags indicating compression and what type.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#33

Post by tehlike »

gopalv wrote: post #16
CodesInChaos wrote: post #5

I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.

> I don't see how that's possible if the cache no longer stores the uncompressed data. Zstd has a seekable format for frames, similar to pigz --independent works. [1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

Yes. This.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#34

Post by colechristensen »

r3trohack3r wrote: post #20

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…

You may or may not be familiar with the content defined chunking family of algorithms like FastCDC.

https://joshleeb.com/posts/chunking.html

https://www.usenix.org/conference/atc16/technical-sessions/p...

On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#35

Post by oefrha »

r3trohack3r wrote: post #20

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#36

Post by jmtulloss »

oefrha wrote: post #35
r3trohack3r wrote: post #20

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…

I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#38

Post by edflsafoiewq »

oefrha wrote: post #35
r3trohack3r wrote: post #20

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…

It was the second most reported problem on the 2025 Rust survey, after compilation times.

Re: We could save petabytes of cache storage with Zstandard and Pingora

#40

Post by chickenbig »

Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck.

It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.

Post reply on HN