Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

161–170 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#161

Post by castis »

erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

1 scenario: if you run a website that has a lot of static content (multiple GB of images, css, js, etc) and you dont want your http server to be responsible for serving that content then you give it all to s3 and let them serve it for you.

Re: Summary of the Amazon S3 Service Disruption

#162

Post by duskwuff »

M_Grey wrote: post #103
jerf wrote: post #73

Earlier quoted context omitted.

I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!

I can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!

Quantum computing: giving humans the unprecedented ability to make every possible error at once

Re: Summary of the Amazon S3 Service Disruption

#163

Post by simonebrunozzi »

CobrastanJorji wrote: post #142
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.

Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.

Re: Summary of the Amazon S3 Service Disruption

#164

Post by fixermark »

joezydeco wrote: post #19
notheguyouthink wrote: post #9

Earlier quoted context omitted.

Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.

You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :

Were it so easy. Likely more like

"Shut down 73 servers? Are you sure? (y/N)":

"Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)

Re: Summary of the Amazon S3 Service Disruption

#165

Post by SilasX »

agency wrote: post #65
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat

I would go with: "To err is human; to cascade, DevOps."

Re: Summary of the Amazon S3 Service Disruption

#166

Post by tdicola »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

To be fair, the real problem isn't that someone screwed up a playbook or command. The real problem is that a tiny mistake in a command can cause an entire service to be disrupted for hours. That's the problem that needs to be fixed.

Re: Summary of the Amazon S3 Service Disruption

#167

Post by smsm42 »

BurningFrog wrote: post #78
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Or the root cause is a UI that allows mistakes like these.

If you have UI that allows to undeploy 10 servers, it will also allow to undeploy 100 servers. Unless you specifically thought about possibility that there might be lower bound of number of servers, which they obviously didn't before that. It's easy to talk about it after the fact, but nobody is able to predict all such scenarios in advance - there are just too many ways to mess up to have special code for all of them in advance.

Re: Summary of the Amazon S3 Service Disruption

#168

Post by [deleted] »

chrisseaton wrote: post #85
BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

[deleted]

Re: Summary of the Amazon S3 Service Disruption

#169

Post by hinkley »

savanaly wrote: post #91
pinko wrote: post #67

Earlier quoted context omitted.

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just t…

It occurs to me that having to type the English version of the numbers would probably work in this scenario.

   s3-shutdown -c "one hundred fifty"
But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

Re: Summary of the Amazon S3 Service Disruption

#170

Post by praneshp »

losteverything wrote: post #104

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

He doesn't seem to have been a great CEO though[0]. Very very sorry for the shitty link, his Wiki page was bare.

[0]: http://www.cnbc.com/2009/04/30/Portfolios-Worst-American-CEO...

Post reply on HN