Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

211–220 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#211

Post by hellbanner »

mabbo wrote: post #153
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

This is good. And for the software engineers, great. I've heard from people doing the grunt work at amazon -- warehouse staff -- that Amazon incentivises employees to rat out each other for mishandling, late time etc, fostering intense competition

Re: Summary of the Amazon S3 Service Disruption

#212

Post by JohnJamesRambo »

obstinate wrote: post #99
chrisseaton wrote: post #85

Earlier quoted context omitted.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

That's what they claim they will do to ameliorate this. They will build limits into their tools.

Why weren't they there already?

Re: Summary of the Amazon S3 Service Disruption

#213

Post by kevinchen »

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

Re: Summary of the Amazon S3 Service Disruption

#214

Post by imsofuture »

Dangeranger wrote: post #23

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…

Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.

Re: Summary of the Amazon S3 Service Disruption

#216

Post by seanwilson »

fixermark wrote: post #139
seanwilson wrote: post #134

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. I find making errors on production when you think you're on staging are a…

"All teams have a test server. Some teams are fortunate enough to also have a separate production server."

> "All teams have a test server. Some teams are fortunate enough to also have a separate production server."

That's what I meant about the SSH comment. Not every team has the automation or infrastructure that allows you to avoid SSH.

Re: Summary of the Amazon S3 Service Disruption

#217

Post by timdorr »

pmoriarty wrote: post #210
mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Putting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.

I think you have it backwards. The post does not say they will simply be training the problem away. They are putting safeguards into their tooling to prevent the case of a fat finger.

Re: Summary of the Amazon S3 Service Disruption

#218

Post by baq »

pmoriarty wrote: post #210
mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Putting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.

The article leaves little doubt that they didn't know such an event would be so hard to recover from. They knew it wouldn't be easy, but they were surprised by how bad it was.

Re: Summary of the Amazon S3 Service Disruption

#219

Post by colanderman »

smsm42 wrote: post #167
BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

If you have UI that allows to undeploy 10 servers, it will also allow to undeploy 100 servers. Unless you specifically thought about possibility that there might be lower bound of number of servers, which they obviously didn't before that. It's easy to talk about it after the fact, but nobody is able to predict all such scenarios in advance - there are just too many ways to mess up to have special code for all of the…

It's not really a UI issue.

The tool as a whole should incorporate a model of S3. Any action you take through the UI should first be applied to this model, and then the resulting impact analyzed. If the impact is "service goes down", then don't apply the action without raising red flags.

Where I work we use PCS for high availability, and it bugs the heck out of me that a fat-fingered command can bring down a service. PCS knows what the effect of any given command will be, but there's no way (that I know of) to do a "dry run" to see whether your services would remain up afterward.

Re: Summary of the Amazon S3 Service Disruption

#220

Post by sndean »

Exuma wrote: post #190

Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP

I've never been in that sort of situation, but I really do wonder how I'd react. Hopefully in healthy way.

How do surgeons react when they accidentally cut the wrong thing?

Post reply on HN