Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#101

Post by kevin2r »

NikolaeVarius wrote: post #12
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Automation doesn't just allow you to create/fix things faster. It also allows you to break things faster.

We may think that an automated system requires less understanding of it in order to operate it. But from the other point of view, you have to know what you are doing, consequences of even an small change are big.

This is one of the things that happens with windows, getting up a server is so easy, that people believe that they don't have to understand what's under the hood, and then, we get a lot of miss-configuration and operational issues.

Re: Summary of the Amazon S3 Service Disruption

#102

Post by netcraft »

chrisseaton wrote: post #85
BurningFrog wrote: post #78

Earlier quoted context omitted.

Or the root cause is a UI that allows mistakes like these.

Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?

It can validate inputs and not let you enter out of range data. You can know that an answer is wrong without knowing what the right answer is.

Re: Summary of the Amazon S3 Service Disruption

#103

Post by M_Grey »

jerf wrote: post #73
agency wrote: post #65

Earlier quoted context omitted.

To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat

I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!

I can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!

Re: Summary of the Amazon S3 Service Disruption

#104

Post by losteverything »

"I did."

That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990

He was asked who made the mistake.

I can't imagine any CEO now a days making a similar statement.

[1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

Re: Summary of the Amazon S3 Service Disruption

#105

Post by edblarney »

cnorthwood wrote: post #6

I'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.

They have no choice in this situation.

CEO's all over the world just realized that they can't only depend on S3, and they might have to double up on their infrastructure and have a parallel env. on Azure or Google as well.

Re: Summary of the Amazon S3 Service Disruption

#106

Post by gigatexal »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

Re: Summary of the Amazon S3 Service Disruption

#107

Post by erikbye »

School-Cotton wrote: post #36
pm90 wrote: post #25

Earlier quoted context omitted.

Wonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".

One of the positive things about Amazon's culture is that they heavily emphasize blaming broken processes, not blaming people. I doubt the person involved will have any negative consequences beyond embarrassment.

I would be horrified if I learned that Amazon or any other company of such size in any way castigates employees for such very human errs. The guilt (don't beat yourself up) he or she likely feels is bad enough.

Anyway, to me this firstly sounds like a "tool" or command that was too powerful with not enough safeguards. Who knows, the command might even be ambiguous.

Re: Summary of the Amazon S3 Service Disruption

#108

Post by ehsankia »

savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

To me, the more important part would've been the solutions they'll come up with to avoid something like this happening again. Is it going to just be "add a line to the playbook asking the engineer to double check the command" or will they make big changes across the system to prevent things like this happening.

Re: Summary of the Amazon S3 Service Disruption

#109

Post by rkuykendall-com »

tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn’t report any of this on the service status dashboard, because we’re idiots, and the dashboard runs on AWS.

Re: Summary of the Amazon S3 Service Disruption

#110

Post by kevan »

BurningFrog wrote: post #78
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Or the root cause is a UI that allows mistakes like these.

Good UX is important, even for things like scripts. Unfortunately a lot of tech people take pride in working with hard-to-use and error-prone tools.
Post reply on HN