Summary of the Amazon S3 Service Disruption

Post by Piskvorrr » Thu, Mar 02, 2017, 10:14 PM UTC

vocatus_gate wrote: post #258

gigatexal wrote: post #106

Earlier quoted context omitted.

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

I once brought down our entire production XenServer cluster group by issuing a "shutdown now" in the wrong SSH window. Needless to say it was a bad feeling watching Nagios go crazy and realizing what had just happened.

    root@baz # shutdown now
    W: molly-guard: SSH session detected!
    Please type in hostname of the machine to shutdown: foo
    Good thing I asked; I won't shutdown baz ...

Post by ghurtado » Thu, Mar 02, 2017, 10:16 PM UTC

jerf wrote: post #73

agency wrote: post #65

Earlier quoted context omitted.

To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat

I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!

Post by zamalek » Thu, Mar 02, 2017, 10:24 PM UTC

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Post by mabbo » Thu, Mar 02, 2017, 10:27 PM UTC

hellbanner wrote: post #211

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

This is good. And for the software engineers, great. I've heard from people doing the grunt work at amazon -- warehouse staff -- that Amazon incentivises employees to rat out each other for mishandling, late time etc, fostering intense competition

Post by kondro » Thu, Mar 02, 2017, 10:31 PM UTC

CWuestefeld wrote: post #178

St-Clock wrote: post #120

I am unpleasantly surprised that they do not mention why services that should be unrelated to S3 such as SES were impacted as well and what they are doing to reduce such dependencies. From a software development perspective, it makes sense to reuse S3 and rely on it internally if you need object storage, but from an ops perspective, it means that S3 is now a single point of failure and that SES's reliability will alw…

services that should be unrelated to S3 such as SES were impacted I don't think this is particularly surprising. I'd already pretty much assumed that, e.g., a package of code for a Lambda function would be housed in an S3 bucket somewhere. What's really surprising to me is how many of those buckets appear to live in US-EAST-1, and aren't able to keep functioning in a catastrophe by failing over to a different region.