Earlier quoted context omitted.
Hindsight is 20/20.
Because human error isn't foreseeable? Or a disgruntled employee?
Because there are limits to engineering resources even at Amazon.
Summary of the Amazon S3 Service Disruption
271–280 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#272> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Sounds like AWS needs Spinnaker for easy rollbacks! https://news.ycombinator.com/item?id=13776456
Re: Summary of the Amazon S3 Service Disruption
#273Earlier quoted context omitted.
I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.
Or have it pull from two sources, one local (S3) and one remote (GCE or whatever), and make a hard positive from either source signal "down." Otherwise the page would be down if just the remote source were down.
Precisely.
"we have changed the SHD administration console to run across multiple AWS regions."
Re: Summary of the Amazon S3 Service Disruption
#274Earlier quoted context omitted.
The point is that at eleven nines, you're entering the realm of very rare/unlikely events that will also affect durability. In other words, there's a lack of humility about "unknown unknowns".
but amazon doesn't offer eleven 9s of availability. I don't think anybody serious does, so arguing how silly eleven 9s of availability is is kind of pointless. The SLA is only four 9s of availability.
Not even four 9's - they only trigger SLA credits when they dip below 3 9's.
Re: Summary of the Amazon S3 Service Disruption
#275Earlier quoted context omitted.
> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.
Hm, based on the description, I would be surprised if they could fat-finger that.
Re: Summary of the Amazon S3 Service Disruption
#276Earlier quoted context omitted.
But that still allows you to make a mistake - by pressing y when that's the wrong thing to do.
There's a balance to be struck. I'd say number of hoops you have to jump through to do something should scale with the potential impact of an operation. That said, the only way to completely prevent mistakes is to make the tool unable to do anything at all. (Or to encode every possible meaning of the word "mistake" in your software. If you could do that, you would probably get a Nobel prize for it.)
In a program I wrote I make the user manually type "I AGREE" (case-sensitive) in a prompt before continuing, just to avoid situations where people just tap "y" a bunch of times.
Re: Summary of the Amazon S3 Service Disruption
#277I think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-)
Shameless plugs (authored months ago): http://tuxlabs.com/?p=380 - How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi-cloud segway...second article) http://tuxlabs.com/?p=430 - AWS, Google Cloud, Azure and the singularity of the future Internet
Re: Summary of the Amazon S3 Service Disruption
#278I think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-)
Shameless plugs (authored months ago): http://tuxlabs.com/?p=380 - How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi-cloud segway...second article)
http://tuxlabs.com/?p=430 - AWS, Google Cloud, Azure and the singularity of the future Internet
Re: Summary of the Amazon S3 Service Disruption
#279Earlier quoted context omitted.
> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
I've heard (and sometimes pushed) this rhetoric before, but something should be well understood before it's automated. Things that happen very rarely should be backed with a playbook + well exercised general monitoring and tools. This puts human discretion in front of the tools' use and makes sure ops is watching for any secondary effects. Ops grimorae can gather disparate one offs into common and tested tools but th…
To me that sounds like development and testing (i.e. figuring out what the steps are). Once you have that it should be automated fully.
Too often people will put up with the, "well, we only do this once a month so it's not worth automating". Literally, I script everything now, just in simple bash... if I type a command, I stick it into a script, and then run the script. Over time you go back and modify said script to be better, eventually this turns into more substantive application. At a certain point, around the time that you have more than one loop, are trying to do things based on different error scenarios, it's probably time to turn to rewriting it in another language.
The simplest thing this does for me, is guarantee that all the parameters needed are valid and present before continuing.
Re: Summary of the Amazon S3 Service Disruption
#280Earlier quoted context omitted.
Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.
Hm, based on the description, I would be surprised if they could fat-finger that.
playbooks can take arguments: http://stackoverflow.com/questions/30662069/how-can-i-pass-v...
So, fat-fingering something is imminently possible.