Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

391–400 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#391

Post by majewsky »

beberlei wrote: post #154
bluejekyll wrote: post #138

Earlier quoted context omitted.

> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.

Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.

I don't think it means "playbook" in the Ansible sense. The dictionary (i.e. Wikipedia) definition of "playbook" is "a document defining one or more business process workflows aimed at ensuring a consistent response to situations commonly encountered during the operation of the business", and that's how I know it.

At $work, certain types of frequently-occurring alerts have playbooks that document how the alert in question can be diagnosed and how known causes can be remedied. Something like "Look at Grafana dashboard X. If metric Y is doing this and that thing, the cause is Z. Log on to box 16 and systemctl restart the foo.service."

Re: Summary of the Amazon S3 Service Disruption

#392

Post by flamedoge »

mabbo wrote: post #153
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

lol what part of "these things should be done proactively and tested over and over in CI" does not make sense to management?

Re: Summary of the Amazon S3 Service Disruption

#393

Post by flamedoge »

joezydeco wrote: post #19
notheguyouthink wrote: post #9

Earlier quoted context omitted.

Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.

You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :

Too easy. should make u run NP-hard solver to confirm.

Re: Summary of the Amazon S3 Service Disruption

#394

Post by swalsh »

magd wrote: post #42

Oh, that interview question. “Tell me about something you broke in your last job"

I once changed a piece of code that was referenced by every page on the customer facing site (10's millions of visits a day) to use a new function that someone had previously written (and was called on 1 page in the site). I mistakenly didn't really look to closely at the implementation of the function, and didn't realize how badly it's caching strategy was designed. When this code was deployed it instantly caused a thundering herd on our cache servers bringing the site down for about ~40 seconds.

I felt TERRIBLE about it.

Re: Summary of the Amazon S3 Service Disruption

#395

Post by draw_down »

imsofuture wrote: post #214
Dangeranger wrote: post #23

So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…

Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.

I couldn't believe that when I saw it in that incident report. Just crazy.

Re: Summary of the Amazon S3 Service Disruption

#396

Post by _ao789 »

vaishaksuresh wrote: post #323
geodel wrote: post #254

Earlier quoted context omitted.

A lot of IT vendors I have worked with, they all were CMM/CMMi level 5. But the crappiness in their work development/process/deployment etc make me wonder if all their efforts go in attaining those certifications as oppose to doing something better.

As someone who worked for an IT vendor with certification and as someone who was part of the certification team at another place, I can assure you that you're right. The certification is more for the organization/unit and the people working do not realize what they are for. Another thing that usually becomes a problem is the rigidity of the certification. Saying you need X, Y and Z documented is easy, but it doesn't…

Thank you for adding this comment. I am glad there are more people out there that aren't afraid to be honest about some of the nonsense 'follow the process no matter what' stuff that I have experienced over the years.

Re: Summary of the Amazon S3 Service Disruption

#398

Post by jacquesm »

Bartweiss wrote: post #240
mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes.

Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit).

This usually then caused the customer to recant on how he/she never meant for anybody to get fired...

It was a funny solution and we got away with it for years, for one because it was pretty rare to get customers that mad to begin with and for another because Frans never wrote any blog posts about it ;)

But I was always waiting for that call from the labor board to ask why we fired someone for who there was no record of employment.

Re: Summary of the Amazon S3 Service Disruption

#399

Post by jessaustin »

logicallee wrote: post #377
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

serious question - why did no one ever accidentally launch and nuke a city, with thousands of nuclear warheads able to do so on short notice? like, AWS presumably puts a lot more redundancy in, and yet with all that effort comes up this far short. Why? It has a huge amount of brainpower all set up so that this never ever happens. Whatever works for the military, can't they adopt those actual best practices?

When I think about questions like these, I recall the Anthropic Principle. Perhaps on lots of planets, intelligent life ceased at the beginning of the Atomic Age. Here we are seven decades (several generations!) in, and we're still alive! The numerator on the odds almost doesn't matter, when you never get to see the denominator. Now that we're finding all these planets, perhaps we ought to start looking for nuclear extinction events? They probably wouldn't leave lasting evidence, but if they're common enough they wouldn't need to...

Actually the accounts I've read seem to indicate that most missile operators simply decided they would never launch no matter what. God bless them, for that.

Re: Summary of the Amazon S3 Service Disruption

#400

Post by mulmen »

mabbo wrote: post #153
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Agreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened?

I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment.

I could be mis-remembering that but it's important that these lessons be applied across the whole company (at least AWS) so it would be a bigger mark against AWS if this is a result of similar tooling to what caused a previous outage.

Post reply on HN