Some people may benefit from the academic exercise: How many caches are utilized in serving a typical website? (Assume a LAMP-like stack; if your number is less than about seven keep thinking)
Downsides of Caching
11–20 of 36 posts
Re: Downsides of Caching
#12The article can probably be succinctly summarized as "Premature optimization is the root of all evil". Most of the authors points are valid, in that caching adds more complexity.
That said, caching is absolutely critical to almost every piece of software ever. Even if you explicitly caching isn't used, a wide variety of caches are likely still being depending upon including CPU caching (L1, L2, L3), OS filesystem caching, DNS caching, ARP caching, etc etc.
Caching certainly adds complexity but it's also one of the best patterns for solving a wide range of performance problems. I would recommend developers spend more time learning and understanding the complexities so that they can make use of caching correctly and without applying it as a premature optimization.
Re: Downsides of Caching
#13I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…
I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.
Re: Downsides of Caching
#14Caching is also bad in distributed systems, because by definition you're creating tail latency: the cache miss case. In a distributed system, you're more likely to hit the worst case in one component, so the cache may not buy you any end user benefit. It might just make performance more difficult to debug. A cache can still be useful if to reduce load and increase capacity... but latency becomes more complex.
That's kinda weird reasoning. Are you saying there's no benefit to an improvement of median latency, if the tail latency remains long? I would disagree. I also would point out that not all systems that can benefit from a cache are latency-sensitive.
Re: Downsides of Caching
#15I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…
I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.
In that sort of environment, I wouldn't be surprised if your app's internal cache ended up paged out anyway...
Re: Downsides of Caching
#16We need smart tools that can automatically make programs utilize a cache. That way, we can have the best of both worlds.
Re: Downsides of Caching
#17Earlier quoted context omitted.
I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.
In that sort of environment, I wouldn't be surprised if your app's internal cache ended up paged out anyway...
I don't allow swap on my machines, and I mlock executable pages, so I'd personally be surprised if anything was paged out.
Re: Downsides of Caching
#18I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…
I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.
Your proposed solution (read from disk on startup and never again) is really a memory-backed data store, not a cache. Caches can miss.
But let's analyze your example. If disk reads take tens of seconds and memory usage is high enough to purge the kernel's disk cache, nothing can save you. Had your process read in everything at the start, it would be using even more memory. Given the same load, one of two things will happen:
1. If you have swap enabled, parts of your process's memory will be swapped-out. Accessing "memory" in this case would cause a page fault and tens of seconds of delay.
2. If you have swap disabled, the OOM-killer will reap your process. When it respawns, it's going to read lots of stuff from disk... and disk reads take tens of seconds. Oops.
Even if an application-level data cache improved performance on heavily-loaded shared hosts, the added costs of software development and maintenance far exceed the cost of better hardware. Hardware is cheap. Developers are expensive.
Re: Downsides of Caching
#19Earlier quoted context omitted.
I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.
Your proposed solution (read from disk on startup and never again) is really a memory-backed data store, not a cache. Caches can miss. But let's analyze your example. If disk reads take tens of seconds and memory usage is high enough to purge the kernel's disk cache, nothing can save you. Had your process read in everything at the start, it would be using even more memory. Given the same load, one of two things will…
Here's an example. You have a 100MB C++ executable that needs 4GB for its own various purposes and 20GB of data that it's serving. The machine has 64GB of memory. If you allocate 24.1GB of memory to the container for this service, disable swap, and mlock the binary and the data files, nothing will go wrong.
On the same machine is a batch process which is reading a 1TB file and writing another 1TB file. If your serving process was reliant on the OS page cache, it would find that its pages were routinely evicted in favor of the batch process.
You're right about swap, that's why only a crank would enable swap. The moment at which swap was a reasonable solution was already behind us 20 years ago.
Re: Downsides of Caching
#20The article can probably be succinctly summarized as "Premature optimization is the root of all evil". Most of the authors points are valid, in that caching adds more complexity. That said, caching is absolutely critical to almost every piece of software ever. Even if you explicitly caching isn't used, a wide variety of caches are likely still being depending upon including CPU caching (L1, L2, L3), OS filesystem cac…
What bugs me is not so much caching as redundant caching.
I've seen applications that have 5 redundant caches, if not more (on-disk cache, OS cache, VM OS cache, stdlib cache, programmer-visible cache). And then you end up killing the actually-important caches (CPU caches, etc) from the amount of redundant copying required...