21–30 of 535 posts
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
I wouldn't want to be the person who wrote the wrong command! Sheesh.
That person is the least likely to make that mistake again!
On a more serious note, if you've never done something like this, you haven't had enough interesting projects.
I've had a decent career and I still managed to:
* re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one
* rename all the Jenkins jobs on the server to the same name, thus deleting hundreds of Jenkins jobs in one fell swoop
"Let him who is without sin cast the first stone" and all that :)
> Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.
If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this mistake is amazing and competent and just had one bad moment.
While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same.
I wouldn't want to be the person who wrote the wrong command! Sheesh.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey
I'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey
http://techblog.netflix.com/2011/07/netflix-simian-army.html
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
I think you meant "Chaos Monkey" [1]. [1] https://github.com/Netflix/chaosmonkey
> Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone. We want to verify that our services automatically re-balance to the functional availability zones without user-visible impact or manual intervention.