Summary of the Amazon S3 Service Disruption

Post by halomru » Thu, Mar 02, 2017, 9:45 PM UTC

jegoodwin3 wrote: post #335

djsumdog wrote: post #222

Earlier quoted context omitted.

Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this." Tests and configuration scripts don't preven…

I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?

Post by bpchaps » Thu, Mar 02, 2017, 9:47 PM UTC

bluejekyll wrote: post #279

inlined wrote: post #204

Earlier quoted context omitted.

I've heard (and sometimes pushed) this rhetoric before, but something should be well understood before it's automated. Things that happen very rarely should be backed with a playbook + well exercised general monitoring and tools. This puts human discretion in front of the tools' use and makes sure ops is watching for any secondary effects. Ops grimorae can gather disparate one offs into common and tested tools but th…

To me that sounds like development and testing (i.e. figuring out what the steps are). Once you have that it should be automated fully. Too often people will put up with the, "well, we only do this once a month so it's not worth automating". Literally, I script everything now, just in simple bash... if I type a command, I stick it into a script, and then run the script. Over time you go back and modify said script to…

  seq 1 10 | xargs -P10 -I{} ssh $host-{} hostname

Post by mabbo » Thu, Mar 02, 2017, 10:05 PM UTC

antaviana wrote: post #243

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

I presume this is well entrenched in the Amazon culture. Jeff Bezos once said: "Good intentions never work, you need good mechanisms to make anything happen"

Post by mabbo » Thu, Mar 02, 2017, 10:08 PM UTC

jordache wrote: post #268

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

that's just a public statement. how do you whether the individual was reprimanded

Post by romaniv » Thu, Mar 02, 2017, 10:10 PM UTC

perlgeek wrote: post #305

pinko wrote: post #67

Earlier quoted context omitted.

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.