Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

241–250 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#241

Post by saalweachter »

gigatexal wrote: post #106
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

Fat fingers are just nature's way of making sure you test your back-up & restore procedures periodically :-)

Re: Summary of the Amazon S3 Service Disruption

#243

Post by antaviana »

mabbo wrote: post #153
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

I presume this is well entrenched in the Amazon culture.

Jeff Bezos once said: "Good intentions never work, you need good mechanisms to make anything happen"

Re: Summary of the Amazon S3 Service Disruption

#244

Post by user15672 »

aestetix wrote: post #224

It sounds like this can be mitigated by making sure everything is run in dry run mode first, and for something mission critical, getting it double-checked by someone before removing the dry run constraint. It's good practice in general, and I'm kind of astonished it's not part of the operational procedures in AWS, as this would have quickly been caught and fixed before ever going out to production.

I'm not sure you've understood what the problem is. They were removing some servers from a group. This isn't something that get's a dry run, not without spinning up the entire AWS infrastructure. It also wouldn't have helped a jot, since the issue came about after an employee executing a playbook made a typo.

There's no way this could have been mitigated with a dry run. They're mitigating it in future by putting more aggressive safeguards in their tooling, which is the correct way to mitigate this sort of issue.

Re: Summary of the Amazon S3 Service Disruption

#245

Post by djhworld »

pwthornton wrote: post #202

This is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.

I highly doubt this claim, humans make mistakes regardless of the control method.

In this particular case the scripts didn't have adequate protections in place, but that's the benefit of hindsight

Re: Summary of the Amazon S3 Service Disruption

#246

Post by ceejayoz »

djsumdog wrote: post #228
kevinchen wrote: post #213

Earlier quoted context omitted.

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

You have to eat your own dogfood, if you're trying to sell a service product. That being said, a better solution would be to stand up a separate infrastructure just for status pages.

"Except for status pages" is a perfectly reasonable exception to "eat your own dogfood".

Re: Summary of the Amazon S3 Service Disruption

#247

Post by TeMPOraL »

chrisseaton wrote: post #189
brianpan wrote: post #182

Earlier quoted context omitted.

A good UI should be able to help, especially in critical situations. I imagine Amazon will consider something like this: > The dosage you ordered is an order of magnitude greater than the dosage most commonly ordered for this medicine. Continue? y/n

But that still allows you to make a mistake - by pressing y when that's the wrong thing to do.

There's a balance to be struck. I'd say number of hoops you have to jump through to do something should scale with the potential impact of an operation.

That said, the only way to completely prevent mistakes is to make the tool unable to do anything at all.

(Or to encode every possible meaning of the word "mistake" in your software. If you could do that, you would probably get a Nobel prize for it.)

Re: Summary of the Amazon S3 Service Disruption

#248

Post by Bartweiss »

simonebrunozzi wrote: post #163
CobrastanJorji wrote: post #142

Earlier quoted context omitted.

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.

I think it's a little better than that, actually.

It says that their ideal-case failure rate is 11 nines; that's how much you should lose to known, lasting issues like machines failing and cutting over.

Amazon's actual SLA offers 2 nines and 3 nines as the credit thresholds. So they're stating the reliability of their known system, and the rest is for events like this.

Re: Summary of the Amazon S3 Service Disruption

#250

Post by Bartweiss »

chrisseaton wrote: post #85
BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

Fat-finger implies you made your mistake once. A UI can't stop you from setting out to do the wrong thing, but it can make it astronomically unlikely to do a different action than the one you intended.

Simple example: I have a git hook which complains at me if I push to master. If I decide "screw you, I want to push to master", it can't assess my decision, but it easily fixes "oops, I thought I was on my branch".

Post reply on HN