Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

311–320 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#311

Post by perlgeek »

dap wrote: post #100

> Removing a significant portion of the capacity caused each of these systems to require a full restart. I'd be interested to understand why a cold restart was needed in the first place. That seems like kind of a big deal. I can understand many reasons why it might be necessary, but that seems like one of the issues that's important to address.

Possibly a consensus algorithm that refuses writes when it detects itself in a minority, because it think it's in the smaller part of a split-brain scenario.

In this case, throwing away and then re-provisioning the split-off nodes is a viable approach.

Re: Summary of the Amazon S3 Service Disruption

#312

Post by andrewguenther »

Dangeranger wrote: post #23

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…

Nothing is going to happen to the engineer that did this, other than embarrassment and probably a couple jokes once sufficient time has passed. Amazon has a strong culture around blaming the process, not the person. The failure here wasn't that the engineer ran the command, it was that the engineer was able to run the command.

Source: AWS employee

Re: Summary of the Amazon S3 Service Disruption

#313

Post by rickcnagy »

hinkley wrote: post #169
savanaly wrote: post #91

Earlier quoted context omitted.

By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just t…

It occurs to me that having to type the English version of the numbers would probably work in this scenario. s3-shutdown -c "one hundred fifty" But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.

Re: Summary of the Amazon S3 Service Disruption

#315

Post by snewman »

hollander wrote: post #309
snewman wrote: post #131

Earlier quoted context omitted.

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…

Testing a full zone test is only possible when they have a new zone available, unused. I bet they do these test, and they now have a new scenario to test.

They also probably have one or more test regions where they could perform a test like this. But it's presumably not at nearly the same scale as us-east-1, the region affected by this incident. And to a considerable extent the problem was one of scale. The writeup makes the recovery sound fairly straightforward; but due to the sheer size of S3 in this region, it took hours for the system to come back up, which was apparently unexpected.

(Nit: this incident affected a region, not a zone. us-east-1 is a region, which is divided into zones us-east-1a, us-east-1b, etc. S3 operates on regions.)

Re: Summary of the Amazon S3 Service Disruption

#316

Post by gist »

losteverything wrote: post #104

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

> "I did."

To me that is just another example of 'caring theater'. Whereby carefully crafted PR responses [1] appear to take responsibility in a 'buck stops here' kind of way. The truth is it is unreasonable in many cases for the top person to be able to prevent any and all errors. If you try and make everything perfect with no mistakes you would never make any money (and of course it's not even possible).

[1] ie 'our customers safety and security is of the utmost importance to us'.

Re: Summary of the Amazon S3 Service Disruption

#317

Post by seansmccullough »

beberlei wrote: post #154
bluejekyll wrote: post #138

Earlier quoted context omitted.

> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.

Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.

It's probably a page on the internal Wiki that the S3 team follows for that particular task. Most the actual steps are probably automated, but it sounds more like a checklist.

I used to follow runbooks/playbooks written on the internal wiki when I worked at Amazon.

Re: Summary of the Amazon S3 Service Disruption

#318

Post by Dangeranger »

andrewguenther wrote: post #312
Dangeranger wrote: post #23

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…

Nothing is going to happen to the engineer that did this, other than embarrassment and probably a couple jokes once sufficient time has passed. Amazon has a strong culture around blaming the process, not the person. The failure here wasn't that the engineer ran the command, it was that the engineer was able to run the command. Source: AWS employee

That's encouraging to hear. I am glad that Amazon has the right target in it's sights for correction.

Re: Summary of the Amazon S3 Service Disruption

#319

Post by camperman »

agency wrote: post #65
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat

In #devops is turtle all way down but at bottom is perl script - DevOps Borat
Post reply on HN