"People make mistakes all the time...the problem was that our systems that were designed to recognize and correct human error failed us." [1]
Summary of the Amazon S3 Service Disruption
111–120 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#112Earlier quoted context omitted.
Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?
It can validate inputs and not let you enter out of range data. You can know that an answer is wrong without knowing what the right answer is.
Possibly the values are all with range. It was just that this operation only worked on elements that were a subset. No amount of validation will catch that error.
You could feedback a clarification, but if that happens too often nobody will double check it after they have seen it over and over.
Re: Summary of the Amazon S3 Service Disruption
#113Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…
I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."
I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations.
I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the command, or there's an approval step, or whatever.
Re: Summary of the Amazon S3 Service Disruption
#114Earlier quoted context omitted.
That person is the least likely to make that mistake again!
No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundr…
As I keep saying, there are people who screw up big time and people who are too scared to touch the system. I managed to gun down three productive clusters by deploying NTP by accident along with a tiny change. Kerpow, 12 minutes of downtime, full network outage due to DHCP and such. Great fun.
Re: Summary of the Amazon S3 Service Disruption
#115Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something. It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove…
To me, the more important part would've been the solutions they'll come up with to avoid something like this happening again. Is it going to just be "add a line to the playbook asking the engineer to double check the command" or will they make big changes across the system to prevent things like this happening.
I'm interested in something like that, too! I've got a few destructive Ansible commands that I check a few dozen times before running. But There's always a chance that I'm tired/distracted/whatever and run something silly anyway. Typically I put in things like config test checks and prompts, but damn it's scary how much power I have with this Ansible setup. I definitely don't want to be in this AWS position.
Re: Summary of the Amazon S3 Service Disruption
#116> While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. This is the bit that'd worry me most; you'd think they'd be testing this.
A complete restart of the index subsystem would require downtime. Note: they are not saying those servers have never been restarted - it's highly likely they get restarted regularly. But, a complete restart of the index subsystem implies that you shut everything down first and restart it all at once, which is what was forced to happen two days ago.
Re: Summary of the Amazon S3 Service Disruption
#117Re: Summary of the Amazon S3 Service Disruption
#118Are customers going to receive any kind of future credit because of this? Would be a nice band-aid after such a hard smack on the head.
You can make a claim to receive credit based on their SLA: https://aws.amazon.com/s3/sla/
You shouldn't have to make a claim - this should be handled by them automatically.
Re: Summary of the Amazon S3 Service Disruption
#119I wouldn't want to be the person who wrote the wrong command! Sheesh.
They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.
The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!")
There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UPDATE without a WHERE) - but given what I know about Amazon's internal practices (i.e. the brutality) it wouldn't surprise me if they did terminate the unfortunate operator - not because they want to, but because AWS simply has too many large-scale customers who would demand immediate action like that.
Re: Summary of the Amazon S3 Service Disruption
#120I am unpleasantly surprised that they do not mention why services that should be unrelated to S3 such as SES were impacted as well and what they are doing to reduce such dependencies.
From a software development perspective, it makes sense to reuse S3 and rely on it internally if you need object storage, but from an ops perspective, it means that S3 is now a single point of failure and that SES's reliability will always be capped by S3's reliability. From a customer perspective, the hard dependency between SES and S3 is not obvious and is disappointing.
The whole internet was talking about S3 when the AWS status dashboard did not show any outage, but very few people mentioned other services such as SES. Next time we encounter errors with SES, should we check for hints of S3 outage before everything else? Should we also check for EC2 outage?