Summary of the Amazon S3 Service Disruption • Hacker News Board

Re: Summary of the Amazon S3 Service Disruption

#431

Post by TeMPOraL » Fri, Mar 03, 2017, 2:14 AM UTC

askmike wrote: post #417

TeMPOraL wrote: post #402

Earlier quoted context omitted.

It's indeed a real problem. Hell, I myself am habituated to logins and passwords for frequently used dialog boxes, and so just two days ago I tried to log in on my work's JIRA account using test credentials for an app we're developing... For securing very dangerous commands, I'd recommend asking the user to retype a phrase composed of random words, or maybe a random 8-character hexadecimal number - something that's d…

I think that even if someone can't memorize the exact characters, they'll memorize the task of having to type over the characters. Better would be to never ask for confirmation except in the worst of worst cases.

That's what I meant in my original comment when I wrote that "number of hoops you have to jump through to do something should scale with the potential impact of an operation". Harmless operations - no confirmation. Something that could mess up your work - y-or-n-p confirmation. Something that could fuck up the whole infrastructure - you'd better get ready to retype a mix of "I DO UNDERSTAND WHAT I'M JUST ABOUT TO DO" and some random hashes.

Re: Summary of the Amazon S3 Service Disruption

#432

Post by patio11 » Fri, Mar 03, 2017, 2:20 AM UTC

justin_oaks wrote: post #418

jacquesm wrote: post #398

Earlier quoted context omitted.

What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…

It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions." It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees.

Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

Re: Summary of the Amazon S3 Service Disruption

#433

Post by vitalus » Fri, Mar 03, 2017, 2:32 AM UTC

awj wrote: post #322

Bartweiss wrote: post #240

Earlier quoted context omitted.

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

Yeah indeed. You know who the one person at Amazon is that I'd expect to never fat finger a sensitive command ever ever again? The guy who managed to fat finger S3 on Tuesday. Firing him over this mistake is worse than pointless, it offers absolution to every other developer and system that helped cause this event.

I wouldn't merely call this a fat finger or typo - it's quite possible that the usage of the tool itself was so nefarious that mistakes would be impossible to avoid, given the complexity of its inputs.

Based on Amazon's decision to improve the tooling such that this category of error would be (hopefully) impossible to reproduce, I would lean more towards that being the case.

Re: Summary of the Amazon S3 Service Disruption

#434

Post by dang » Fri, Mar 03, 2017, 2:35 AM UTC

njharman wrote: post #184

Did you read the fucking article? That is EXACTLY what they are doing (among other things).

Please stop posting like this so we don't have to ban you.

https://news.ycombinator.com/newsguidelines.html

We detached this comment from https://news.ycombinator.com/item?id=13776335 and marked it off-topic.

Re: Summary of the Amazon S3 Service Disruption

#435

Post by seanwilson » Fri, Mar 03, 2017, 2:35 AM UTC

eric_h wrote: post #349

seanwilson wrote: post #134

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. I find making errors on production when you think you're on staging are a…

We have a deploy script that does exactly this, unfortunately we've all gotten our muscle memory so trained that most of us type the deploy command, press enter, type yes and press enter before we're ever even prompted. Fortunately in most cases a quick ctrl-c can prevent any actual damage.

Type 'yes'? Make it ask you to type out the deploy location. I think that helps a lot.

Re: Summary of the Amazon S3 Service Disruption

#436

Post by rconti » Fri, Mar 03, 2017, 2:38 AM UTC

magd wrote: post #42

Oh, that interview question. “Tell me about something you broke in your last job"

Paged for "$HOSTNAME is down!"

Usual checks to access $HOSTNAME failed

Rushed to office at 6am before important process was about to run that needed that host.

Plugged in keyboard+monitor, dead screen, nothing.

Physically power-cycled server.

Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down.

Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension.

The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP.

And then I raced to the office just so that I could cause an outage.

Re: Summary of the Amazon S3 Service Disruption

#437

Post by mrep » Fri, Mar 03, 2017, 3:04 AM UTC

jordache wrote: post #268

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

that's just a public statement. how do you whether the individual was reprimanded

I believe Jeff once said something along the lines of "why would I fire an employee that made an honest mistake? I just spent a bunch of money teaching him a lesson"

Re: Summary of the Amazon S3 Service Disruption

#438

Post by losteverything » Fri, Mar 03, 2017, 3:09 AM UTC

pcthrowaway wrote: post #427

losteverything wrote: post #173

Earlier quoted context omitted.

I'll try. It was a dark day. We all watched the news and I recall him saying that. The specific quote I don't remember but it was something like " you can consider that I did." I think he was asked what will happen to the person that caused it and who is that person. Everyone knew right away this had to be human error. Right away. Switches simply had too much redundancy. It was big then and not sure if I can locate a…

Not sure if there's a video, but I found an article with the exact quote: > As far as our customers are concerned, I did it. http://www.upi.com/Archives/1990/01/16/ATT-pinpoints-cause-o...

Thanks. I got the gist at least.

I admired him and his answer at the time. The culture was quite professional and blame really never existed.

Re: Summary of the Amazon S3 Service Disruption

#439

Post by evilDagmar » Fri, Mar 03, 2017, 3:10 AM UTC

hibikir wrote: post #430

Bartweiss wrote: post #287

Earlier quoted context omitted.

Fair enough. I guess what I meant was specifically using training or punishment to combat "momentary lapse" issues. If someone doesn't test their backups, you train them to test backups. If someone lies about testing the backups, maybe you fire them. But if someone trips and shatters the only backup disk, you don't yell at them - you create backups that an instant of clumsiness can't ruin. I did overstate, training i…

Training about testing backups is still a bad idea: Why make someone do a job that purely verification? Those jobs eventually stop getting done, and it's hard to keep people doing them. Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those…

No. You don't make a daily task of testing backups. That would be wrong for precisely the reasons you cite. It's a waste of effort and time, and ignores what the point of testing them is for: ensuring that the procedure still works.

One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. The rest of the year it's only worth keeping an automated process reporting whether or not the things are being made, and people keeping an eye on change management to be sure no changes are made to the known-to-be working process that can break it without the new process incurring an explicit vetting cycle. Gitlab wasn't apparently testing or engaging in monitoring what was supposed to be an automated process. That's where they got burned.

Process monitoring may be boring as hell, but it's seldom wasted effort, and will prevent massive, compounded headaches from bringing operations to a chaotic halt.

Re: Summary of the Amazon S3 Service Disruption

#440

Post by School-Cotton » Fri, Mar 03, 2017, 3:11 AM UTC

keldaris wrote: post #429

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

I don't understand how this is even possible in a company operating on that scale. Granted, I'm a lowly scientific programmer with no clue about running a cloud infrastructure, but I would have imagined that there would be at least a pretense of oversight for destructive commands run in such an environment. A scheme as simple as "any destructive command run on S3 subsystems is automatically run in a dry run form, and…

I thought the same thing before I went into the industry but now that I've been in it for a few years (including two at Amazon), it doesn't surprise me.

I suspect locking everyone down in the way you suggest would cost more in lost productivity (and costs for the infrastructure that would be required for greater auditing, etc.) than is lost in outages like this.