Summary of the Amazon S3 Service Disruption • Hacker News Board

Re: Summary of the Amazon S3 Service Disruption

#251

Post by refulgentis » Thu, Mar 02, 2017, 7:21 PM UTC

mabbo wrote: post #153

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

How laudable is this, really?

I've had the privilege of either working for myself, the company that acquired mine and let me run the dev, or at Google. From that perspective, and what I understand about ops, the rarity is not having the attitude mentioned in the parent.

Re: Summary of the Amazon S3 Service Disruption

#252

Post by TeMPOraL » Thu, Mar 02, 2017, 7:22 PM UTC

colanderman wrote: post #219

smsm42 wrote: post #167

Earlier quoted context omitted.

If you have UI that allows to undeploy 10 servers, it will also allow to undeploy 100 servers. Unless you specifically thought about possibility that there might be lower bound of number of servers, which they obviously didn't before that. It's easy to talk about it after the fact, but nobody is able to predict all such scenarios in advance - there are just too many ways to mess up to have special code for all of the…

It's not really a UI issue. The tool as a whole should incorporate a model of S3. Any action you take through the UI should first be applied to this model, and then the resulting impact analyzed. If the impact is "service goes down", then don't apply the action without raising red flags. Where I work we use PCS for high availability, and it bugs the heck out of me that a fat-fingered command can bring down a service.…

Interesting.

In practice, it would likely be very hard to make a model of your infrastructure to test against, but I can imagine a tool that would run each query against a set of heuristics, and if any flags pop up, it would make you jump through some hoops to confirm. Such a tool should NEVER have an option to silently confirm, and the only way to adjust a heuristic if it becomes invalid should be formally getting someone from an appropriate department to change it and sign off on it.

By the way, this is how companies acquire red tape. It's like scar tissue.

Re: Summary of the Amazon S3 Service Disruption

#253

Post by Eiriksmal » Thu, Mar 02, 2017, 7:23 PM UTC

westernmostcoy wrote: post #135

Take a moment to look at the construction of this report. There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't. You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in…

Interesting observation. Maybe the answer is it that a behemoth like AWS does this because they _can_ get away with it. In contrast to AWS's cascading failures, the GitLab outage was a mere blip. Because they are several orders of magnitude smaller than Amazon, however, they had to be painfully transparent during their actual restore operations and in the post-moterm.

AWS has more implicit trust that this won't happen again, since they've never (I think?) had something like this happen, so just a few lines about fixing the tool that let all the nodes shutdown is enough to restore confidence.

Re: Summary of the Amazon S3 Service Disruption

#254

Post by geodel » Thu, Mar 02, 2017, 7:23 PM UTC

liquidise wrote: post #227

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

This is the major basis of the CMM Levels [1]. At higher levels of maturity and necessity, systems and processes are designed to increasingly prevent errors from reaching a production environment. Amazon is taking the right approach here. The fact that a system as complex and important as S3 can be taken down is a failure of the system, not the person who took it down accidentally. 1. https://en.wikipedia.org/wiki/Ca…

A lot of IT vendors I have worked with, they all were CMM/CMMi level 5. But the crappiness in their work development/process/deployment etc make me wonder if all their efforts go in attaining those certifications as oppose to doing something better.

Re: Summary of the Amazon S3 Service Disruption

#255

Post by neurostimulant » Thu, Mar 02, 2017, 7:24 PM UTC

JohnJamesRambo wrote: post #212

obstinate wrote: post #99

Earlier quoted context omitted.

That's what they claim they will do to ameliorate this. They will build limits into their tools.

Why weren't they there already?

Hindsight is 20/20.

Re: Summary of the Amazon S3 Service Disruption

#256

Post by kwisatzh » Thu, Mar 02, 2017, 7:25 PM UTC

illumin8 wrote: post #116

rsynnott wrote: post #54

> While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. This is the bit that'd worry me most; you'd think they'd be testing this.

A complete restart of the index subsystem would require downtime. Note: they are not saying those servers have never been restarted - it's highly likely they get restarted regularly. But, a complete restart of the index subsystem implies that you shut everything down first and restart it all at once, which is what was forced to happen two days ago.

Why can't the index subsystem itself have a backup then? When the primary subsystem is being restarted/rebuilt, the secondary takes over.

Re: Summary of the Amazon S3 Service Disruption

#257

Post by TeMPOraL » Thu, Mar 02, 2017, 7:28 PM UTC

jrgifford wrote: post #179

phonon wrote: post #94

Earlier quoted context omitted.

"Just pull the power!"

"We've got redundant power - you can't!"

"So take the emergency chainsaw, and go cut the Internet cables."

Re: Summary of the Amazon S3 Service Disruption

#258

Post by vocatus_gate » Thu, Mar 02, 2017, 7:28 PM UTC

gigatexal wrote: post #106

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

I once brought down our entire production XenServer cluster group by issuing a "shutdown now" in the wrong SSH window. Needless to say it was a bad feeling watching Nagios go crazy and realizing what had just happened.

Re: Summary of the Amazon S3 Service Disruption

#259

Post by hunterjrj » Thu, Mar 02, 2017, 7:28 PM UTC

sndean wrote: post #220

Exuma wrote: post #190

Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP

I've never been in that sort of situation, but I really do wonder how I'd react. Hopefully in healthy way. How do surgeons react when they accidentally cut the wrong thing?

Oh man, finally a chance to plug the book "Do No Harm: Stories of Life, Death, and Brain Surgery" by Henry Marsh.

Marsh chronicles his career, and includes (at least) one story about the slip of the scalpel and the... result.

Highly recommended.

Re: Summary of the Amazon S3 Service Disruption

#260

Post by jstanley » Thu, Mar 02, 2017, 7:30 PM UTC

HeyLaughingBoy wrote: post #174

pavel_lishin wrote: post #150

Earlier quoted context omitted.

As I understand it, those guarantees don't mean that the service will actually stay up for the given number of 9s; it's that you'll be reimbursed monetarily if and when they go down.

Kinda the same thing, though. I mean, from my perspective there's no substantive difference between me saying "this service will stay up 99.xx% of the time and me buying insurance to pay you for the 0.xx% of the time I might fail. The alternative is that I use the insurance to pay my legal fees when you sue me for not meeting my uptime guarantees.

It's not the same thing. The Amazon service might only be costing you $100/mo, but if it goes down the cost to your business might be millions. They'll reimburse you the $100, not the millions.