A decade of major cache incidents at Twitter
1–10 of 27 posts
Re: A decade of major cache incidents at Twitter
#2Major incidents aside, I always think that cache-related bugs are some of the most likely to go undetected since if you don't test for them end-to-end, they're really not that easy to spot & diagnose.
An article sticking around too long on the home page. Semi-stale data creeping into your pipeline. Someone's security token being accepted post-revocation. All really hard to spot unless (1) you're explicitly looking, or (2) manure hits the fan.
Re: A decade of major cache incidents at Twitter
#3What I find most interesting in this is the pseudo detective story of hunting down disappearing post-mortem and "lessons learned" documentation. Optimistically we'd hope that perhaps the older systems no longer reflect the existing systems in any meaningful way (possibly as the org structures and/or software stacks shift and change) and they're no longer relevant.
I'd imagine most lost knowledge is not an explicit decision however which means such historical scenarios / documentation / ... are just lost as part of business. Lost knowledge is the default for companies.
Twitter is likely better than most given their documentation is all digital and there exist explicit processes to catalogue such incidents. I'd also be curious to see how much of this knowledge has been implicitly exported to their open source codebases.
Re: A decade of major cache incidents at Twitter
#4I remember reading Facebooks caches had a dedicated standby set of “gutter” servers that would take over a failure quickly (otherwise inactive and unused) that was an interesting mitigation for some failure scenarios.
Re: A decade of major cache incidents at Twitter
#5"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton
Re: A decade of major cache incidents at Twitter
#6Required reading for all of the "I could code up Twitter in a weekend" -types.
The long listen queue -> multiple queued up retries feedback loop is a classic: https://datatracker.ietf.org/doc/html/rfc896 TCP/IP "congestion collapse" and the 1986 Internet meltdown [various sources]
Re: A decade of major cache incidents at Twitter
#7"There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton https://martinfowler.com/bliki/TwoHardThings.html
There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.
Re: A decade of major cache incidents at Twitter
#8Required reading for all of the "I could code up Twitter in a weekend" -types. The long listen queue -> multiple queued up retries feedback loop is a classic: https://datatracker.ietf.org/doc/html/rfc896 TCP/IP "congestion collapse" and the 1986 Internet meltdown [various sources]
It ultimately depends at what scale. I think most people to be fair are talking about building a clone of what was present back in the mid 2000s. You could build a twitter clone that could handle a few hundred users in a few weeks with modern tech stacks.
The fact that there are at least 3 twitter clones that are less well put together with a decent amount of users handling in the load proves that it is possible.
Re: A decade of major cache incidents at Twitter
#9Required reading for all of the "I could code up Twitter in a weekend" -types. The long listen queue -> multiple queued up retries feedback loop is a classic: https://datatracker.ietf.org/doc/html/rfc896 TCP/IP "congestion collapse" and the 1986 Internet meltdown [various sources]
It ultimately depends at what scale. I think most people to be fair are talking about building a clone of what was present back in the mid 2000s. You could build a twitter clone that could handle a few hundred users in a few weeks with modern tech stacks. The fact that there are at least 3 twitter clones that are less well put together with a decent amount of users handling in the load proves that it is possible.
“A few weeks” sounds a lot longer than a weekend, and I’d also consider the history: Twitter itself was built quickly using a modern stack. Rails was highly productive, the problem is that the concept of the service makes scaling non-trivial. We have more RAM and SSDs now so you could get further but those aren’t magic.
Re: A decade of major cache incidents at Twitter
#10Earlier quoted context omitted.
It ultimately depends at what scale. I think most people to be fair are talking about building a clone of what was present back in the mid 2000s. You could build a twitter clone that could handle a few hundred users in a few weeks with modern tech stacks. The fact that there are at least 3 twitter clones that are less well put together with a decent amount of users handling in the load proves that it is possible.
“A few weeks” sounds a lot longer than a weekend, and I’d also consider the history: Twitter itself was built quickly using a modern stack. Rails was highly productive, the problem is that the concept of the service makes scaling non-trivial. We have more RAM and SSDs now so you could get further but those aren’t magic.
I agree. I was playing devil's advocate to an extent. At the time Rails was the first modern MVC framework as we understand it.
> Rails was highly productive,the problem is that the concept of the service makes scaling non-trivial.
Didn't they rewrite everything in PHP during the last 2000s due to Rails at the time just not being able to scale?