61–70 of 535 posts
This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.
Oh, that interview question. “Tell me about something you broke in your last job"
TLDR; Someone on the team ran a command by mistake that took everything down. Good, detailed description. It happens. Out of all of Amazon's offerings, I still love S3 the most.
Earlier quoted context omitted.
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
Kudos to the dev team for that, I think most people wouldn't own that kind of issue
Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
We have geo-distributed systems. Load balancing and automatic failover. We agonize over edge cases that might cause issues. We build robust systems.
At the end of he day reliability -- a lot like security -- is most affected by the human factor.
Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…
TLDR; Someone on the team ran a command by mistake that took everything down. Good, detailed description. It happens. Out of all of Amazon's offerings, I still love S3 the most.
The only thing I read in there and go "hmmm" is that it took quite that long for the S3 service to recover, and that the status page wasn't hosted on someone that doesn't have an S3 dependency. That's just a plain "doh" moment :)
This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.