Summary of the Amazon S3 Service Disruption

Post by [deleted] » Thu, Mar 02, 2017, 5:30 PM UTC

urda wrote: post #21

conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey

Post by [deleted] » Thu, Mar 02, 2017, 5:30 PM UTC

urda wrote: post #21

conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey

Post by saisundar » Thu, Mar 02, 2017, 5:30 PM UTC

urda wrote: post #21

conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey

Post by mcrider » Thu, Mar 02, 2017, 5:31 PM UTC

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Post by yellow_postit » Thu, Mar 02, 2017, 5:31 PM UTC

Scramblejams wrote: post #13

aorloff wrote: post #11

What's missing is addressing the problems with their status page system, and how we all had to use Hacker News and other sources to confirm that US East was borked.

No, this is addressed: >We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.