Remember folks, automate your systems but never forget to add sanity checks.
Summary of the Amazon S3 Service Disruption
71–80 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#72Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…
I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."
then you need to "sudo take them all offline now!"
Re: Summary of the Amazon S3 Service Disruption
#73> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat
I've long said something like "To err is human. To fuck up a million times in a second you need a computer."
I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!
Re: Summary of the Amazon S3 Service Disruption
#74Seems like something like Chaos monkey should have been able to predict and mitigate a issue like this. Im actually curious if anyone uses it at all- Curious if anyone in here at a large company (over 500 employees) has it deployed or not.
Re: Summary of the Amazon S3 Service Disruption
#75Are customers going to receive any kind of future credit because of this? Would be a nice band-aid after such a hard smack on the head.
Re: Summary of the Amazon S3 Service Disruption
#76[deleted]
Re: Summary of the Amazon S3 Service Disruption
#77Earlier quoted context omitted.
I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."
then you need to "sudo take them all offline now!"
"We can't because we put a safeguard in for that last week!"
Re: Summary of the Amazon S3 Service Disruption
#78> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Or the root cause is a UI that allows mistakes like these.
Re: Summary of the Amazon S3 Service Disruption
#79I wouldn't want to be the person who wrote the wrong command! Sheesh.
That person is the least likely to make that mistake again!
Reminds me of the story about Bob Hoover.[1]
He took off in his piston engine plane, only to lose power during the climb and was forced to make a crash landing. It turned out the airplane was fueled with jet fuel instead of regular gasoline (the ground crewman mistakenly thought the plane was a turbo prop).
Instead of yelling at or firing the ground crewman, Hoover had this to say[2]:
"There isn't a man alive who hasn't made a mistake.
But I'm positive you'll never make this mistake again.
That's why I want to make sure that you're the only one
to refuel my plane tomorrow. I won't let anyone else
on the field touch it."
Re: Summary of the Amazon S3 Service Disruption
#80For as much as people jumped all over Gitlab last month, this seems remarkably similar in terms of preparedness for accidental and unanticipated failure.