Summary of the Amazon S3 Service Disruption • Hacker News Board

Re: Summary of the Amazon S3 Service Disruption

#231

Post by joelthelion » Thu, Mar 02, 2017, 7:00 PM UTC

pyre wrote: post #160

tlack wrote: post #26

Earlier quoted context omitted.

And apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.

To put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.

> Perhaps they should have tested this in a simulated environment

What makes you think they didn't?

Re: Summary of the Amazon S3 Service Disruption

#232

Post by nemacol » Thu, Mar 02, 2017, 7:01 PM UTC

duskwuff wrote: post #162

M_Grey wrote: post #103

Earlier quoted context omitted.

I can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!

Quantum computing: giving humans the unprecedented ability to make every possible error at once

It will be fucked, not fucked, neither, and both... until we look. I feel bad for the poor bastard that has to look...

Re: Summary of the Amazon S3 Service Disruption

#233

Post by idlewords » Thu, Mar 02, 2017, 7:07 PM UTC

pyre wrote: post #160

tlack wrote: post #26

Earlier quoted context omitted.

And apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.

To put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.

Moreover, beyond a certain scale, it becomes really hard to simulate.

Re: Summary of the Amazon S3 Service Disruption

#234

Post by Twirrim » Thu, Mar 02, 2017, 7:11 PM UTC

rmason wrote: post #149

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

So they're going to build a complex system to correct possible user command line errors. That new system itself will introduce possible errors. Wouldn't an administrative GUI have been much simpler to implement overall?

No, they're introducing safeguards to an already established system. It's not even that complex, for that matter.

Re: Summary of the Amazon S3 Service Disruption

#235

Post by pmoriarty » Thu, Mar 02, 2017, 7:11 PM UTC

losteverything wrote: post #104

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

What are the real consequences for a CEO saying that? It's not like he's going to get fired or have his stock options revoked. If anything, people are going to praise him for taking ownership like that, as you did. Virtually no matter what he does, I'd bet someone in his position is going to be very comfortable for the rest of his life.

I'd be far more impressed if a low-level employee who's whole family depended on his job and who stood a good chance of getting fired admitted a serious mistake.

Re: Summary of the Amazon S3 Service Disruption

#236

Post by ceejayoz » Thu, Mar 02, 2017, 7:12 PM UTC

pmoriarty wrote: post #210

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Putting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.

> Maybe they need something like the procedure that's used in missile silos...

Probably a bad example. The system was a pain in the ass, so they went and circumvented some of its restrictions.

http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-u...

> Those in the U.S. that had been fitted with the devices, such as ones in the Minuteman Silos, were installed under the close scrutiny of Robert McNamara, JFK's Secretary of Defence. However, The Strategic Air Command greatly resented McNamara's presence and almost as soon as he left, the code to launch the missile's, all 50 of them, was set to 00000000.

> Oh, and in case you actually did forget the code, it was handily written down on a checklist handed out to the soldiers.

Re: Summary of the Amazon S3 Service Disruption

#237

Post by xtracto » Thu, Mar 02, 2017, 7:13 PM UTC

savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

But just imagine the "ooooh shit" moment of this person.

Something similar happened to us, when an Engineer deleted part of our production database with a single command. Fortunately, we could reconstruct it from backups and replication logs.

Re: Summary of the Amazon S3 Service Disruption

#238

Post by jordigh » Thu, Mar 02, 2017, 7:13 PM UTC

hueving wrote: post #69

certifiedloud wrote: post #14

This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.

The term "playbook" just originates from sports as a documentation of what you are going to do for for a specific task (i.e. a "play" in sports).

I'm pretty sure the sports meaning comes from the theathre meaning. Ansible has other bits of theatre metaphor in it, such as "roles" and "scripts".

Re: Summary of the Amazon S3 Service Disruption

#239

Post by vorpalhex » Thu, Mar 02, 2017, 7:13 PM UTC

westernmostcoy wrote: post #135

Take a moment to look at the construction of this report. There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't. You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in…

Emails seem to be going out. I got one a while ago. I suspect this was an initial response geared towards the general audience and a more specific technical response will be forthcoming.

Re: Summary of the Amazon S3 Service Disruption

#240

Post by Bartweiss » Thu, Mar 02, 2017, 7:14 PM UTC

mabbo wrote: post #153

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.