201–210 of 535 posts
Earlier quoted context omitted.
I think the article says he was authorized? Either way, at worst, he got fired.
I assume if they were unauthorized they would be fired, but since they were authorized shit happens. "Fire you? I just spent $10 million training you!"
Earlier quoted context omitted.
You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :
Were it so easy. Likely more like "Shut down 73 servers? Are you sure? (y/N)": "Seventy-three? Wow, I hadn't realized our system grew that much. Probably a new backend dependency got added that I'm not familiar with yet; I'll look into it later." (Y)
Source: I'm was the one the one they would call for our team... usually at 4:00 AM because one of our team members (which was also frequently me) didn't document something correctly.
Earlier quoted context omitted.
Automation tends to make those kinds of errors worse rather than better. Perhaps more infrequent and of a different nature than before, but screwing up an automated action cascades much, much faster than a human initiated one. As a result, you have to watch things a good deal closer and build in more and tighter safe guards. For instance: https://thenextweb.com/shareables/2014/05/16/emory-universit... Note: Automatio…
> established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
Dear Amazon: please lease a $25/month dedicated server to host your status page on.
Earlier quoted context omitted.
Or the root cause is a UI that allows mistakes like these.
If you have UI that allows to undeploy 10 servers, it will also allow to undeploy 100 servers. Unless you specifically thought about possibility that there might be lower bound of number of servers, which they obviously didn't before that. It's easy to talk about it after the fact, but nobody is able to predict all such scenarios in advance - there are just too many ways to mess up to have special code for all of the…
Earlier quoted context omitted.
It's one of the reasons that silly guarantees like "twelve 9s of reliability" are meaningless. There are humans here. "Accidental human mishap" is gonna happen sometimes, and when it happens it's probably gonna affect a lot of data. Heck, at around 7 or 8 nines you have to account for the possibility that your operations team will decide that all your data is a vicious pack of timberwolves and needs to be defeated.
Note that's durability not reliability . You might not be able to get at it with every request (I think 99.99% is the target) but it'll still be there if you try again later.
In other words, there's a lack of humility about "unknown unknowns".
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…
Rough guide:
CT = cost of 1 full scale test with necessary infrastructure and labor costs added up
CF = amount of money paid out in SLA claims + subjective estimate of business lost due to reputation damage etc
PF = estimate of probability of this event happening in a given year
if PF * CF > CT, then you run such a test at least once a year. Think of such an expense as an insurance premium.
What Netflix does with their simian army is amortize the cost of doing the test across millions of tests per year and the extra design complications arising from having to deal with failures that often.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Is mere extra training the right solution here?
Maybe they need something like the procedure that's used in missile silos:
Not allowing the shutdown system to function at all without the explicit authorization of least two people.