Live data from Hacker News

A decade of major cache incidents at Twitter

danluu.com

21–27 of 27 posts

Re: A decade of major cache incidents at Twitter

#21
post #5

"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton https://martinfowler.com/bliki/TwoHardThings.html

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.

That's the one I use all the time.

Re: A decade of major cache incidents at Twitter

#22
post #19

Earlier quoted context omitted.

Yes, I agree with that. Containers though, as far as some of the advantages; we have been using chroots for deployment since the early 2000s which is not the same but deployment/compatibility wise it was pretty good. It allows you to have the same small Linux image and deployment everywhere as well and you could move most zipped images from machine to machine with vastly different kernels. I still use chroots now on…

I used chroots, too, and it was useful but much harder to maintain than a container. Automation wasn't impossible, of course, but that was also complicated by concerns about bloating each chroot with copies of all of the system libraries & config files. I used PHP in that era. It could be faster but then you're in the classic developer productivity tradeoff between, say, hand-coded SQL calls versus using an ORM, etc.…

Hahah yeah ; I did a lot of hosting then and everyone insisted on register_globals. I still run stuff for clients that depends on it to this day.

And maybe the rails perf was comparable to the larger pho frameworks in some cases, but php was really much easier to scale in our experience. We hosted millions of sites and the python/ruby ones were generally dramas when they got serious (for that time) traffic.

Re: A decade of major cache incidents at Twitter

#23
post #5

"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton https://martinfowler.com/bliki/TwoHardThings.html

There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.

I prefer the less obvious version of this one:

> There are only three hard things in Computer Science: cache invalidation and naming things.

Re: A decade of major cache incidents at Twitter

#24
These big incidents involving 'big cache' are fun to read about. Years ago I had to deal with a bunch of cache issues over a short time, but they were all minor incidents with minor uses of cache (simple memoization, storing stuff in maps on attributes of java singletons, browser local storage). Still, I made a checklist of questions to ask thenceforth on any proposal or implementation of a cache in a doc or code review. A bunch of them are just focused on actually paying attention to what your keys are made of and how invalidation works (or if you even can invalidate, or if it's even needed). I think for 'big cache' questions I should just refer to this blog post and ask "what's the risk of these issues?"

Re: A decade of major cache incidents at Twitter

#25
post #19

Earlier quoted context omitted.

I used chroots, too, and it was useful but much harder to maintain than a container. Automation wasn't impossible, of course, but that was also complicated by concerns about bloating each chroot with copies of all of the system libraries & config files. I used PHP in that era. It could be faster but then you're in the classic developer productivity tradeoff between, say, hand-coded SQL calls versus using an ORM, etc.…

Hahah yeah ; I did a lot of hosting then and everyone insisted on register_globals. I still run stuff for clients that depends on it to this day. And maybe the rails perf was comparable to the larger pho frameworks in some cases, but php was really much easier to scale in our experience. We hosted millions of sites and the python/ruby ones were generally dramas when they got serious (for that time) traffic.

You could still write bad code, of course, but deploying mod_php sure was easy. People could still write bad code (thinking here of someone who processed a database join in a foreach loop rather than learning how to use a WHERE constraint) but I do miss that level of install simplicity, at least until I remember what it was like dealing with incompatible versions or reconciling configuration in multiple places.

Re: A decade of major cache incidents at Twitter

#27
post #3

What I find most interesting in this is the pseudo detective story of hunting down disappearing post-mortem and "lessons learned" documentation. Optimistically we'd hope that perhaps the older systems no longer reflect the existing systems in any meaningful way (possibly as the org structures and/or software stacks shift and change) and they're no longer relevant. I'd imagine most lost knowledge is not an explicit de…

What you've said is, in my opinion, likely to be a difference between the technology companies that become tomorrow's infrastructure and the ones that disappear (even if it takes decades).

As you say, the default tendency in many companies when failures occur is information-loss. That can be attributed to using too many communication tools, cultural expectations that problems should be hidden, silo'd or disparate documentation stores, or lack of process.

Intentional, open, thorough and replicated note-taking with cross-references before, during and after incidents can create radically different environments which allow for querying, recovery and improvement regardless of failure mode(s). Kudos to Dan for moving in that direction with these writeups (and to you for raising the subtext).

Post reply on HN