Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Downsides of Caching

msol.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Downsides of Caching

#21

Post by bigiain »

thrownaway2424 wrote: post #13
ggreer wrote: post #8

I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…

I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.

Can I suggest that if you've got problems including "stalls for tens of seconds on disk reads" you are almost certainly better off directing available resources towards fixing your hosting problems, rather than going down the cache rabbit hole on a hosting platform that's not really suitable for production use?

(With caveats for zero resource projects of course, but even for those I strongly suspect for many people paying $5 or $10 per month for "less crap hosting" is probably a better solution that prematurely optimising by adding caching and all it's inherent complexity to a fundamentally broken platform)

Re: Downsides of Caching

#22

Post by glandium »

thrownaway2424 wrote: post #19
ggreer wrote: post #18

Earlier quoted context omitted.

Your proposed solution (read from disk on startup and never again) is really a memory-backed data store, not a cache. Caches can miss. But let's analyze your example. If disk reads take tens of seconds and memory usage is high enough to purge the kernel's disk cache, nothing can save you. Had your process read in everything at the start, it would be using even more memory. Given the same load, one of two things will…

Here's an example. You have a 100MB C++ executable that needs 4GB for its own various purposes and 20GB of data that it's serving. The machine has 64GB of memory. If you allocate 24.1GB of memory to the container for this service, disable swap, and mlock the binary and the data files, nothing will go wrong. On the same machine is a batch process which is reading a 1TB file and writing another 1TB file. If your servin…

If only O_STREAMING had made it to the kernel... https://lwn.net/Articles/12100/

Re: Downsides of Caching

#23

Post by Gibbon1 »

thrownaway2424 wrote: post #14
chubot wrote: post #4

Caching is also bad in distributed systems, because by definition you're creating tail latency: the cache miss case. In a distributed system, you're more likely to hit the worst case in one component, so the cache may not buy you any end user benefit. It might just make performance more difficult to debug. A cache can still be useful if to reduce load and increase capacity... but latency becomes more complex.

That's kinda weird reasoning. Are you saying there's no benefit to an improvement of median latency, if the tail latency remains long? I would disagree. I also would point out that not all systems that can benefit from a cache are latency-sensitive.

I read some scribbling by some nerd working on distributed systems. The problem he mentioned is when you take a task and parallelize it, and then hand off the pieces to a bunch of workers, you aren't done until the last worker finishes. In that case long tail latencies can bite you rather hard. If 99 out of a hundred workers finish their bit in 50-100us and one of them stalls out for 10ms, you gained nothing over a single worker.

Re: Downsides of Caching

#24

Post by ggreer »

thrownaway2424 wrote: post #19
ggreer wrote: post #18

Earlier quoted context omitted.

Your proposed solution (read from disk on startup and never again) is really a memory-backed data store, not a cache. Caches can miss. But let's analyze your example. If disk reads take tens of seconds and memory usage is high enough to purge the kernel's disk cache, nothing can save you. Had your process read in everything at the start, it would be using even more memory. Given the same load, one of two things will…

Here's an example. You have a 100MB C++ executable that needs 4GB for its own various purposes and 20GB of data that it's serving. The machine has 64GB of memory. If you allocate 24.1GB of memory to the container for this service, disable swap, and mlock the binary and the data files, nothing will go wrong. On the same machine is a batch process which is reading a 1TB file and writing another 1TB file. If your servin…

In that example, I'm pretty sure forgoing containers and mlock would result in similar performance while using less memory. Process startup time would also be significantly improved. (If there's such high contention for disk I/O, reading 20GB on startup is going to take a very long time.)

The kernel's page cache eviction strategy is smarter than naïve LRU. On the first read, a page is placed in the inactive file list. If it's read again, it's moved to the active file list. Pages in the inactive file list are purged before the active file list.[1] So large sequential reads may cause disk contention, but they won't massacre the file cache.

This I/O situation isn't uncommon. Consumer systems also have big batch jobs that can pollute file caches: large copies, rsyncs, backup software (Déjà Dup, Time Machine, etc). They don't solve this with containers, limits, and mlock()ing. Some programs add a couple calls to fadvise(), using the FADV_NOREUSE or FADV_DONTNEED flags.[2] But for the most part, doing nothing yields excellent performance. Operating systems are pretty good at their job.

1. https://www.kernel.org/doc/gorman/html/understand/understand...

2. This is handy for applications like bittorrent, where multiple reads of the same page are possible, but caching isn't desired.

Re: Downsides of Caching

#25

Post by thrownaway2424 »

bigiain wrote: post #21
thrownaway2424 wrote: post #13

Earlier quoted context omitted.

I don't think that's very good advice in a heavily-loaded shared hosting environment. A disk read could easily stall for tens of seconds, just because the kernel whimsically decided to throw out the cache (or because your server crowded its memory container). I actually don't want any server touching a disk while it's serving. Everything should be read before service begins and never again.

Can I suggest that if you've got problems including "stalls for tens of seconds on disk reads" you are almost certainly better off directing available resources towards fixing your hosting problems, rather than going down the cache rabbit hole on a hosting platform that's not really suitable for production use? (With caveats for zero resource projects of course, but even for those I strongly suspect for many people p…

There's nothing you or I can do about the trend. They put more and more cores into a machine and the same number of disks (current-model Xeon servers have 72 threads and 1 or 2 disks), which guarantees that, at some point, the disk is highly oversubscribed.

Re: Downsides of Caching

#26

Post by viraptor »

ggreer wrote: post #8

I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…

I wouldn't say "never". Let's say you have a local file with 1e6 words. The file can be updated at some point. Your service gets a word in a request and needs to return "is this word in the list".

Do you really want to read the file every time at request comes in? No, you're going to read it once and store it in an indexed set for quick lookup. You just cached a local data file.

It's about the benefit vs. not caching. Not about local/remote.

Re: Downsides of Caching

#27

Post by ggreer »

viraptor wrote: post #26
ggreer wrote: post #8

I agree with pretty much everything in this post, though I would add one more thing. It's not so much a downside of caching as a misuse: Application-level caches should never cache local data. Cache network responses. Cache the results of computations. Don't cache files or disk reads. Operating systems already implement disk caches, and they do a better job of it than you. That's in addition to a modern computer's nu…

I wouldn't say "never". Let's say you have a local file with 1e6 words. The file can be updated at some point. Your service gets a word in a request and needs to return "is this word in the list". Do you really want to read the file every time at request comes in? No, you're going to read it once and store it in an indexed set for quick lookup. You just cached a local data file. It's about the benefit vs. not caching…

Your example is quite valid, and I would probably implement something similar to solve the same problem. But it's not a cache. Caches can miss. Caches have a replacement policy. If it contains the complete, authoritative copy of the data, it's a memory-backed data store.

Re: Downsides of Caching

#28

Post by rileymat1 »

0xcde4c3db wrote: post #7

It occurs to me that sharding shares most of these disadvantages. It avoids the problem of "you no longer read from your source of truth", but the overall complexity and set of failure modes looks strikingly similar. I wonder how many sleepless nights have been caused by combining the two.

I have worked with a couple of systems that used very course grained sharding at the application level. I did not notice these drawbacks. I have not worked with one that did auto sharding on the back end, that might be trickier.

Re: Downsides of Caching

#29

Post by bigiain »

thrownaway2424 wrote: post #25
bigiain wrote: post #21

Earlier quoted context omitted.

Can I suggest that if you've got problems including "stalls for tens of seconds on disk reads" you are almost certainly better off directing available resources towards fixing your hosting problems, rather than going down the cache rabbit hole on a hosting platform that's not really suitable for production use? (With caveats for zero resource projects of course, but even for those I strongly suspect for many people p…

There's nothing you or I can do about the trend. They put more and more cores into a machine and the same number of disks (current-model Xeon servers have 72 threads and 1 or 2 disks), which guarantees that, at some point, the disk is highly oversubscribed.

Sure - the "race to the bottom" for hosting prices inevitably means there's going to be options like GoDaddy offering "a year's worth of webhosting for $5" which clusters 400 WordPress and Drupal sites onto a single RaspberryPi or similar, but you don't _have_ to go there.

I can understand if you're an open source developer who gets paid in Uzbeki Som or Nigerian Naira, the calculation of "do I spend a day or two putting caching in place" versus "do I spend an extra $50 or $100 per year on hosting" might lean very much the other way, but I suspect for the vast majority of HN readers, the prudent approach is "pay a hundred or two dollars a year for hosting before bothering to implement complex caching strategies".

Re: Downsides of Caching

#30

Post by chubot »

thrownaway2424 wrote: post #14
chubot wrote: post #4

Caching is also bad in distributed systems, because by definition you're creating tail latency: the cache miss case. In a distributed system, you're more likely to hit the worst case in one component, so the cache may not buy you any end user benefit. It might just make performance more difficult to debug. A cache can still be useful if to reduce load and increase capacity... but latency becomes more complex.

That's kinda weird reasoning. Are you saying there's no benefit to an improvement of median latency, if the tail latency remains long? I would disagree. I also would point out that not all systems that can benefit from a cache are latency-sensitive.

Not that there's no benefit, but just that it's more complicated in a distributed system.

Certainly caching is vital to many distributed systems, but it has to be done from a systems perspective. In my experience a lot of caches are just slapped on top of individual components without much thought, and without even some basic monitoring of what the hit rate is. I think it helps to actually measure what the cache is doing for you -- but this is more work than adding the cache itself.

And I agree with another poster in that I've seen many systems with caches papering over severe and relatively obvious performance problems in the underlying code.

I was thinking of this Google publication which outlines some problems with latency variability: http://www.barroso.org/publications/TheTailAtScale.pdf

Interestingly they didn't seem to list caches as one of the causes; they list shared resources, cron jobs, queuing, garbage collection, power saving features, etc.

Post reply on HN