Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

401–410 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#401

Post by jessaustin »

pm90 wrote: post #25
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Wonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".

Haha it's funny that a simple misread of "authorized" as "unauthorized" got this comment downvoted to oblivion...

Re: Summary of the Amazon S3 Service Disruption

#402

Post by TeMPOraL »

Piskvorrr wrote: post #372
vocatus_gate wrote: post #276

Earlier quoted context omitted.

In a program I wrote I make the user manually type "I AGREE" (case-sensitive) in a prompt before continuing, just to avoid situations where people just tap "y" a bunch of times.

Habituation is a powerful thing: a safety-critical program used in the 90s had a similar, hard-coded safety prompt (<10 uppercase ASCII characters). Within a few weeks, all elevated permission users had the combination committed to muscle memory and would bang it out without hesitation, just by reflex: "Warning: please confirm these potentially unsaf-" "IAGREE!"

It's indeed a real problem. Hell, I myself am habituated to logins and passwords for frequently used dialog boxes, and so just two days ago I tried to log in on my work's JIRA account using test credentials for an app we're developing...

For securing very dangerous commands, I'd recommend asking the user to retype a phrase composed of random words, or maybe a random 8-character hexadecimal number - something that's different every time, so can't be memorized.

Re: Summary of the Amazon S3 Service Disruption

#403

Post by mulmen »

awj wrote: post #322
Bartweiss wrote: post #240

Earlier quoted context omitted.

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

Yeah indeed. You know who the one person at Amazon is that I'd expect to never fat finger a sensitive command ever ever again? The guy who managed to fat finger S3 on Tuesday. Firing him over this mistake is worse than pointless, it offers absolution to every other developer and system that helped cause this event.

I think the value of making these mistakes is in learning from them and then making sure they can't happen again. Leaving this process in place and just making this guy run the command forever because he screwed it up once would be a much less effective solution than fixing the tooling so it's impossible to do this in the first place. Saying "this guy don't do it again" also offers absolution to everyone else on the team. In a healthy culture only we can fail.

Re: Summary of the Amazon S3 Service Disruption

#405

Post by FormerAMZN »

I'm a little surprised everyone is giving them a pass on this:

    S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected.
I'm biased - so take this with a grain of salt. But, I don't think your fast growth is an excuse for not doing some fairly basic stuff. What other processes have you not restarted in a long time?

Re: Summary of the Amazon S3 Service Disruption

#406

Post by bbcbasic »

ellyagg wrote: post #338
bbcbasic wrote: post #328

Earlier quoted context omitted.

That old chestnut. Is it true?

Is it not true for you? I know that I'm personally good at avoiding the same mistake. I'm also extraordinarily good at avoiding repeating catastrophic mistakes. I generally change my processes in the same way that Amazon is changing their processes to avoid this mistake.

I am not talking about what Amazon are doing, but the concept that the individual wont make the same mistake again, which is what the grandparent is getting that.

He won't make the same mistake because no one makes the same big mistake twice? I wouldn't bank on that alone.

Re: Summary of the Amazon S3 Service Disruption

#407

Post by marvinalone »

logicallee wrote: post #377
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

serious question - why did no one ever accidentally launch and nuke a city, with thousands of nuclear warheads able to do so on short notice? like, AWS presumably puts a lot more redundancy in, and yet with all that effort comes up this far short. Why? It has a huge amount of brainpower all set up so that this never ever happens. Whatever works for the military, can't they adopt those actual best practices?

In 2016, the UK accidentally fired a Trident missile at the US mainland: http://www.dw.com/en/uk-government-covered-up-disastrous-fai...

This one did not carry a warhead. Others do...

Re: Summary of the Amazon S3 Service Disruption

#408

Post by 72deluxe »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

GUIs are really great. They're a recent development in the computing industry that help mitigate this sort of problem. You can even put prompts in that get you to confirm Yes/No to continue.

I think Borland do some RAD systems, and Microsoft have an IDE of sorts on the way too.

EDIT: Please note that this is humour.

Re: Summary of the Amazon S3 Service Disruption

#409

Post by twunde »

Piskvorrr wrote: post #361
vocatus_gate wrote: post #258

Earlier quoted context omitted.

I once brought down our entire production XenServer cluster group by issuing a "shutdown now" in the wrong SSH window. Needless to say it was a bad feeling watching Nagios go crazy and realizing what had just happened.

root@baz # shutdown now W: molly-guard: SSH session detected! Please type in hostname of the machine to shutdown: foo Good thing I asked; I won't shutdown baz ... Surprising to see such a simple protection neglected.

I don't know how well-known molly-guard is, but I've never heard of it before. Definitely enabling it on my servers next week.

Re: Summary of the Amazon S3 Service Disruption

#410

Post by autokad »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

feel sorry for that user, I'd want to hide in a corner
Post reply on HN