[dead]
We could save petabytes of cache storage with Zstandard and Pingora
41–50 of 64 posts
Re: We could save petabytes of cache storage with Zstandard and Pingora
#42Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#43Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#44Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
Some compression libraries have an 'rsync compatibility mode', that plays some games with the block sizes to make it easier to rsync to not have to completely re-transmit a large compressed file because not the whole file changes every time.
I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#45Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
Use a cargo workspace. Helps with both compilation speed and disk usage.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#46I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
> I don't see how that's possible if the cache no longer stores the uncompressed data. Zstd has a seekable format for frames, similar to pigz --independent works. [1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
Speaking of pigz, I've run into pigzpp[1], or rather its paper: "pigzpp: Fast, Parallel, Portable Compression for the Whole Stack"[2].
Turns out we can squeeze quite a bit more compression performance out of DEFLATE - ~10x in certain instances, 2x as a base minimum (read the paper for details).
Re: We could save petabytes of cache storage with Zstandard and Pingora
#47Earlier quoted context omitted.
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big. It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file…
Don't they have nightly for imperfect experiments like this?
Re: We could save petabytes of cache storage with Zstandard and Pingora
#48One thing that bothered me enough to comment on: "transcoding" doesn't seem like the right term in this context. "Encode/decode" is technically correct, but "compress/decompress" would have made the intent much clearer.
Transcoding is generally used within the context of audio or video codecs, to convert a file from one format to another. And usually from one lossy format to another (eg: not raw uncompressed YUV420, YUV422P video or whatever that is stored in a lossless compression format). It's not clear to me why they're using it in this web page.
Re: We could save petabytes of cache storage with Zstandard and Pingora
#49Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck. It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data…
> compression levels -1, 1 or 2
Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
Re: We could save petabytes of cache storage with Zstandard and Pingora
#50Earlier quoted context omitted.
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them. It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone. You may be cleaving the pr…
>user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them).
But I have to admit the idea is clever.