Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

131–140 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#131

Post by snewman »

conorh wrote: post #17

This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.

Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Also, hindsight is 20/20, but before this incident it might have been hard to identify "full-scale restart of the index subsystem" as rising to the top of the list of things to test.

One approach is to try to extrapolate from smaller-scale tests. It would be interesting to know what kinds of disaster testing Amazon does do, and at what scale, and whether a careful reading could have predicted this outcome.

Re: Summary of the Amazon S3 Service Disruption

#132

Post by unit91 »

Johnny555 wrote: post #18
ct0 wrote: post #2

I wouldn't want to be the person who wrote the wrong command! Sheesh.

I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.

I did once, too. Unfortunately, it was company of 4 people so I was also the dev team. :-) Never did it again though (yet).

Re: Summary of the Amazon S3 Service Disruption

#133

Post by erikbye »

prh8 wrote: post #80

For as much as people jumped all over Gitlab last month, this seems remarkably similar in terms of preparedness for accidental and unanticipated failure.

In the case of GitLab weren't they actually close to losing a lot of data? Was any data lost (or chance of loss) in this incident?

Re: Summary of the Amazon S3 Service Disruption

#134

Post by seanwilson »

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

I find making errors on production when you think you're on staging are a big one for similar errors. One of the best things I ever did on one job was to change the deployment script so that when you deployed you would get a prompt saying "Are you sure you want to deploy to production? Type 'production' to confirm". This helped stop several "oh my god, no!" situations when you repeated previous commands without thinking. For cases where you need to use SSH as well (best avoided but not always practical), it helps to use different colours, login banners and prompts for the terminals.

Re: Summary of the Amazon S3 Service Disruption

#135

Post by westernmostcoy »

Take a moment to look at the construction of this report.

There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't.

You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in cloud computing. We should all demand better.

Re: Summary of the Amazon S3 Service Disruption

#136

Post by fixermark »

GalacticCmdr wrote: post #112
netcraft wrote: post #102

Earlier quoted context omitted.

It can validate inputs and not let you enter out of range data. You can know that an answer is wrong without knowing what the right answer is.

Possibly the values are all with range. It was just that this operation only worked on elements that were a subset. No amount of validation will catch that error. You could feedback a clarification, but if that happens too often nobody will double check it after they have seen it over and over.

While you can't prevent user error without preventing user capability, you can (as others have observed) follow some common heuristics to avoid common failure modes.

A confirm step in something as sensitive as this operation is important. It won't stop all user error, but it gives a user about to accidentally turn off the lights on US-EAST-1 an opportunity to realize that's what their command will do.

Re: Summary of the Amazon S3 Service Disruption

#138

Post by bluejekyll »

emeraldd wrote: post #83
ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Automation tends to make those kinds of errors worse rather than better. Perhaps more infrequent and of a different nature than before, but screwing up an automated action cascades much, much faster than a human initiated one. As a result, you have to watch things a good deal closer and build in more and tighter safe guards. For instance: https://thenextweb.com/shareables/2014/05/16/emory-universit... Note: Automatio…

> established playbook

A playbook actually represents a lack of automation for a particular task.

The playbook itself should be automated, with automated tests that validate its correctness.

Re: Summary of the Amazon S3 Service Disruption

#139

Post by fixermark »

seanwilson wrote: post #134

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. I find making errors on production when you think you're on staging are a…

"All teams have a test server. Some teams are fortunate enough to also have a separate production server."

Re: Summary of the Amazon S3 Service Disruption

#140

Post by harryh »

erikbye wrote: post #130

What does everyone use S3 for? I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad…

It's a very easy way to cheaply store static data to be served over HTTP. There are many many many many many many people who want to do that.
Post reply on HN