tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn’t report any of this on the service status dashboard, because we’re idiots, and the dashboard runs on AWS.
Everything except the "We're idiots" part I'd agree with.
Self-hosting your diagnostic tools is an easy mistake to make, and I've seen both startups and large, multi-decade-experienced companies make it.
"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...
Fascinating! Do you know of any source that documents the claim that Robert Allen made this statement?
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail.
The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Automation tends to make those kinds of errors worse rather than better. Perhaps more infrequent and of a different nature than before, but screwing up an automated action cascades much, much faster than a human initiated one. As a result, you have to watch things a good deal closer and build in more and tighter safe guards. For instance: https://thenextweb.com/shareables/2014/05/16/emory-universit... Note: Automatio…
> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.
I keep being reminded of something I read recently that made me feel uneasy about google's cloud spanner [1]: the most important one is that Spanner runs on Google’s private network. Unlike most wide-area networks, and especially the public internet, Google controls the entire network and thus can ensure redundancy of hardware and paths, and can also control upgrades and operations in general. Fibers will still be cu…
Google takes active steps to confirm that their fallback systems for mitigating failures work.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Wonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
The post never once blames human error, but always specifies that the tools were the problem.
The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UP…
Maybe so. I didn't take it that way. I took it as, "nobody panic, we weren't hacked."
It's both. This isn't "our system was compromised by attack;" it's "SNAFU."
Everyone who's had operations experience knows that there will be, as time approaches infinity, more than zero SNAFU. That's why companies offer five nines of uptime, not 100% uptime.
Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…
I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."
Instead of rate limiting you'd be better off making it a two-key operation once you hit X threshold. Jr admin can delete 10, but the boss needs to confirm a deletion of 100.
> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…
And apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.
To put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.