Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

181–190 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#181

Post by SilasX »

BurningFrog wrote: post #78
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Or the root cause is a UI that allows mistakes like these.

Agreed -- a postmortem should cite that deployment goof as the immediate cause, with a contributory cause of "you can goof like this without getting a warning etc".

Re: Summary of the Amazon S3 Service Disruption

#182

Post by brianpan »

chrisseaton wrote: post #85
BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

A good UI should be able to help, especially in critical situations. I imagine Amazon will consider something like this:

> The dosage you ordered is an order of magnitude greater than the dosage most commonly ordered for this medicine. Continue? y/n

Re: Summary of the Amazon S3 Service Disruption

#183

Post by erikbye »

peu4000 wrote: post #171
erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

Database backups, puppet modules, a couple low traffic static sites, assets for one of our projects.

Database backups: with the upload speeds I've seen, completing a backup of a database with hundreds of GB would take a really long time. I don't want that extra load and keeping that connection open forever.

Re: Summary of the Amazon S3 Service Disruption

#185

Post by duskwuff »

gigatexal wrote: post #106
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.

Re: Summary of the Amazon S3 Service Disruption

#186

Post by throwaway4job »

simonebrunozzi wrote: post #163
CobrastanJorji wrote: post #142

Earlier quoted context omitted.

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.

Note they say 'durability' not 'availability'.

Re: Summary of the Amazon S3 Service Disruption

#187

Post by gregtaleck »

tdicola wrote: post #166
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

To be fair, the real problem isn't that someone screwed up a playbook or command. The real problem is that a tiny mistake in a command can cause an entire service to be disrupted for hours. That's the problem that needs to be fixed.

"While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future."

Re: Summary of the Amazon S3 Service Disruption

#188

Post by gigatexal »

duskwuff wrote: post #185
gigatexal wrote: post #106

Earlier quoted context omitted.

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.

hmm, that's kinda cool. I'm in a MS shop and I don't know if SMSS has the same feature. My manager just looked at me and said "welp, go restore the table and be more careful next time." I was a new DBA at the time, still, kinda new.

Re: Summary of the Amazon S3 Service Disruption

#189

Post by chrisseaton »

brianpan wrote: post #182
chrisseaton wrote: post #85

Earlier quoted context omitted.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

A good UI should be able to help, especially in critical situations. I imagine Amazon will consider something like this: > The dosage you ordered is an order of magnitude greater than the dosage most commonly ordered for this medicine. Continue? y/n

But that still allows you to make a mistake - by pressing y when that's the wrong thing to do.
Post reply on HN