Man, I'd really hate to be that guy.
Summary of the Amazon S3 Service Disruption
291–300 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#292Can Amazon take responsibility and offer say 10% discount to all the customers who are spending >$X ?
Re: Summary of the Amazon S3 Service Disruption
#293Ops and Engineering here.
My guts hurt just reading this.
With big failures is never just one thing. There are a series of mistakes, bad choices, and ignorance that lead to a big system wide failures.
Re: Summary of the Amazon S3 Service Disruption
#294> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…
Anyone who works on large complex systems will read this is and go "this was all preventable, but also completely understandable".
I agree with that in general but having your monitoring system be dependent on the thing it monitors is a pretty big goof. It possible that the dependency was very non-obvious and many layers deep, which is more understandable, but still...its pretty fundamental.
Re: Summary of the Amazon S3 Service Disruption
#295Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP
I can imagine being that guy in that exact moment. But I can't imagine being that guy after the event. There will be a constant fear and doubt in my mind. And a constant fear whether others trust me anymore. I couldn't quit because that might make me look bad and I couldn't continue because that might make me look bad.
This guy may actually need therapy to not suffer from some light degree of trauma.
Re: Summary of the Amazon S3 Service Disruption
#296Re: Summary of the Amazon S3 Service Disruption
#297> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Yeh, makes sense to make changes to the system rather than do nothing and just blame someone. Errors happen, it's something you can't avoid.
Re: Summary of the Amazon S3 Service Disruption
#298So this is the second high profile outage in the last month caused by a simple command line mistake. > Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this…
> While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same.
I hadn't thought about how 'prone to forgive' an organization is.
Should there be policies in place as to how big a mistake you are allowed to commit? Something akin to:
```if(log10(cost_of_mistake) >= 5): worker.fire() ```
Re: Summary of the Amazon S3 Service Disruption
#299There are quite a few comments here ignoring the clarity that hindsight is giving them. Apparently the devops engineers commenting here have never fucked up.
Re: Summary of the Amazon S3 Service Disruption
#300Earlier quoted context omitted.
Anyone who works on large complex systems will read this is and go "this was all preventable, but also completely understandable".
I agree with that in general but having your monitoring system be dependent on the thing it monitors is a pretty big goof. It possible that the dependency was very non-obvious and many layers deep, which is more understandable, but still...its pretty fundamental.
The monitoring system was not dependent on the thing it was monitoring.
The website that shows the public results of the monitoring, which is updates only by humans, depended on it.