61–65 of 65 posts
It's extremely surprising me to that something like this - be it with gzip, lz4, or whatever - wasn't implemented at Cloudflare ages ago... And that the task to investigate fell to an intern. These are non-negligible numbers...
It's extremely surprising me to that something like this - be it with gzip, lz4, or whatever - wasn't implemented at Cloudflare ages ago... And that the task to investigate fell to an intern. These are non-negligible numbers...
I suspect it just hasn’t been a priority problem in the past. This could very well have been a premature optimization when compared to getting services up and running reliably.
Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then…
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed…
It was part of the (now defunct) frea project: https://freajs.github.io/frea-website/
Earlier quoted context omitted.
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them. It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone. You may be cleaving the pr…
>user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them. This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them). But I have to admit the idea is clever.
Earlier quoted context omitted.
When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime. Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your…
You are making an argument against caching it at all which is clearly not what they want. So the comparison must be uncompressed storage vs compressed storage. The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits. So compression makes the most sense for something you want to keep a long time that will be acce…