What's missing is addressing the problems with their status page system, and how we all had to use Hacker News and other sources to confirm that US East was borked.
Summary of the Amazon S3 Service Disruption
11–20 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#12> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Automation doesn't just allow you to create/fix things faster. It also allows you to break things faster.
Re: Summary of the Amazon S3 Service Disruption
#13What's missing is addressing the problems with their status page system, and how we all had to use Hacker News and other sources to confirm that US East was borked.
No, this is addressed:
>We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.
Re: Summary of the Amazon S3 Service Disruption
#14This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.
Re: Summary of the Amazon S3 Service Disruption
#15Same as that db1 / db2 for GitLab, naming things is pretty important (E.g production / staging, production-us-east-1-db-560 etc.)
Re: Summary of the Amazon S3 Service Disruption
#16This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.
"playbook" is a relatively common term for "documented step-by-step procedure for specific tasks". Effectively, a script with #!/bin/human at the top.
Re: Summary of the Amazon S3 Service Disruption
#17This part is also interesting:
> While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.
These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post:
> We build our systems with the assumption that things will occasionally fail
Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.
Re: Summary of the Amazon S3 Service Disruption
#18I wouldn't want to be the person who wrote the wrong command! Sheesh.
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
Re: Summary of the Amazon S3 Service Disruption
#19Earlier quoted context omitted.
At these scales it's the fault of the system, not the individual, so hopefully they don't come down hard on them.
Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.
You mean like range checking the input parameters to the command? =)
SHUT DOWN S3? ARE YOU SURE? (y/N) :Re: Summary of the Amazon S3 Service Disruption
#20This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.
"playbook" is a relatively common term for "documented step-by-step procedure for specific tasks". Effectively, a script with #!/bin/human at the top.
Also known as a runbook