Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

261–270 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#261

Post by TeMPOraL »

fixermark wrote: post #141
1_2__3 wrote: post #113

Earlier quoted context omitted.

I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations. I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the comm…

The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input. I've always found that explanation a little threadbare.

I think the hoops you have to jump through should scale with severity. Then if you find yourself writing scripts that screen-scrape interactive output to get around some safeguards, you're doing something very wrong.

Re: Summary of the Amazon S3 Service Disruption

#262

Post by tn13 »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Number of lawyers must have drafted those lines and 5 people including Bezos must have approved those lines.

Those lines are not reflective of what Amazon is but what picture Amazon wants to paint now. They have clarified it is their error and not some hacking attempt. Secondly they have not vilified the engineer in question because already Amazon's culture is a bit of a ??? in public mind.

But they have got it right. Shit happens and this is not the first time it has happened or the last time it will happen. Also it will happen with Microsoft, Google and everyone else.

May be we will build even better technologies that will rely on two different cloud providers instead of 1.

Re: Summary of the Amazon S3 Service Disruption

#263

Post by problems »

Bartweiss wrote: post #240
mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

This is true in some cases, but not when mitigations aren't practiced properly - it's not the fat fingered user who should be fired or retrained, but the designer or maintainer of the system that allowed it to become a serious issue.

Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just turned into a major multi-day one.

That's a problem and one which could be preventable with training (or, arguably, firing and hiring). Maintaining your backups properly should be someone's duty, designing and testing systems to minimize impact of user error should be too.

Re: Summary of the Amazon S3 Service Disruption

#264

Post by EdSharkey »

It's curious they needed to "remove capacity" to cure a slow billing problem.

Is that code for a "did you try to reboot the system?" kind of troubleshooting?

It sounds to me like the authorized engineer sent a command to reboot/reimage a large swath of the S3 infrastructure.

Re: Summary of the Amazon S3 Service Disruption

#265

Post by sebringj »

If Amazon were a guy, he'd be a standup guy. This is a very detailed and responsible explanation. S3 has revolutionized my businesses and I love that service to no end. These problems happen very rarely but I may have backups just in case using nginx proxy approach at some point and because S3 is so good, everyone seems to adopt their API so its just a matter of a switch statement. Werner can sweat less. Props.

I would add, it would be awesome if there was a simulation environment, beyond just a test environment that simulated servers outside requesting in, before a command was allowed to run onto production, like a robot deciding this, then could mitigate this, kind of like TDD on steriods if they don't have that already.

Re: Summary of the Amazon S3 Service Disruption

#267

Post by TeMPOraL »

idlewords wrote: post #205

"we have changed the SHD administration console to run across multiple AWS regions." Dear Amazon: please lease a $25/month dedicated server to host your status page on.

If big cloud companies would host their status pages on each other's servers, that would be... actually pretty cool.

Re: Summary of the Amazon S3 Service Disruption

#268

Post by jordache »

mabbo wrote: post #153
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

that's just a public statement. how do you whether the individual was reprimanded

Re: Summary of the Amazon S3 Service Disruption

#269

Post by kilroy123 »

xtracto wrote: post #237
savanaly wrote: post #52

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…

But just imagine the "ooooh shit" moment of this person. Something similar happened to us, when an Engineer deleted part of our production database with a single command. Fortunately, we could reconstruct it from backups and replication logs.

I really feel for this person. I would feel extremely shitty.

Re: Summary of the Amazon S3 Service Disruption

#270

Post by usernametbd »

Exuma wrote: post #190

Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP

I can imagine being that guy in that exact moment. But I can't imagine being that guy after the event. There will be a constant fear and doubt in my mind. And a constant fear whether others trust me anymore. I couldn't quit because that might make me look bad and I couldn't continue because that might make me look bad.
Post reply on HN