121–130 of 535 posts
Earlier quoted context omitted.
Or the root cause is a UI that allows mistakes like these.
Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?
no, because developers still produce code with bugs.
Earlier quoted context omitted.
Is it possible to build a UI that will not allow you to make a mistake? If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?
You don't have to know what would be a mistake. E.g. if the tool is used most of the time to operate on a small set of servers, you have some extra confirmation or command-line option for removing a large set. That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
"People make mistakes all the time...the problem was that our systems that were designed to recognize and correct human error failed us." [1] [1] http://articles.latimes.com/1999/oct/01/news/mn-17288
Something that wasn't addressed -- there seems to be an architectural issue with ELB where ELBs with S3 access logs enabled had instances fail ELB health checks, presumably while the S3 API was returning 5XX. My load balancers in us-east-1 without access logs enabled were fine throughout this event. Has there been any word on this?
I think it comes down to how important your ELB logs are -- if they are important enough that you don't want to allow traffic without logs (i.e. if you're using them for some sort of auditing/compliance), then failing when it can't write the logs seems like the right choice.
> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…
Earlier quoted context omitted.
Agreed. They also seemed to acknowledge that in the post, as they mentioned improving the tool to not allow such destructive options.
You mean like range checking the input parameters to the command? =) SHUT DOWN S3? ARE YOU SURE? (y/N) :
SHUT DOWN S3? ARE YOU SURE? ("I'm absolutely positive."/n) :
Earlier quoted context omitted.
Wonder what happened to the poor slob who did that. He/She was unauthorized AND caused a pretty serious outage... EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
I think the article says he was authorized? Either way, at worst, he got fired.
"Fire you? I just spent $10 million training you!"
Earlier quoted context omitted.
They just spent millions of dollars training that person to never make this mistake again, should definitely keep them around.
The wording of the article implies Amazon is shifting the blame entirely on the individual who typo'd: they indemnify themselves with "an authorized S3 team member using an established playbook..." ("don't blame us, our process is perfect!") There are process-fixes for this, such as requiring a two-person rule when at a production shell and modifying tooling to detect potentially unintentional commands (e.g. a SQL UP…
I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad as well.