Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

341–350 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#341

Post by gmac »

jegoodwin3 wrote: post #335
djsumdog wrote: post #222

Earlier quoted context omitted.

Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this." Tests and configuration scripts don't preven…

I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?

NASA doesn't make these sorts of mistakes

https://www.wired.com/2010/11/1110mars-climate-observer-repo...

Re: Summary of the Amazon S3 Service Disruption

#342

Post by tacon »

duskwuff wrote: post #185
gigatexal wrote: post #106

Earlier quoted context omitted.

Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.

FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.

Is there a collection of data safety tips like this somewhere? I never knew this existed. What else am I missing?

Re: Summary of the Amazon S3 Service Disruption

#343

Post by hinkley »

rickcnagy wrote: post #313
hinkley wrote: post #169

Earlier quoted context omitted.

It occurs to me that having to type the English version of the numbers would probably work in this scenario. s3-shutdown -c "one hundred fifty" But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.

I was in fact thinking of the --force problem which is why I went another way.

Re: Summary of the Amazon S3 Service Disruption

#345

Post by BurningFrog »

JohnJamesRambo wrote: post #212
obstinate wrote: post #99

Earlier quoted context omitted.

That's what they claim they will do to ameliorate this. They will build limits into their tools.

Why weren't they there already?

Maybe they were, but they missed this one thing?

Re: Summary of the Amazon S3 Service Disruption

#346

Post by ASalazarMX »

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Somehow, it's reassuring to see how 45 years later, the rm -rf class of problem still persists.

Re: Summary of the Amazon S3 Service Disruption

#347

Post by chiph »

kevinchen wrote: post #213
jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

Or to host it on a static .html page that gets rewritten every 60 seconds or so by an external process, running on physical servers. Minimal stack, so minimal attack surface.

Re: Summary of the Amazon S3 Service Disruption

#348

Post by el_benhameen »

CephalopodMD wrote: post #339
perlgeek wrote: post #307

Earlier quoted context omitted.

Also notably missing is the "we will automatically refund all affected customers" line that we'd expect from somebody who wants to provide excellent service. A graphical illustration of the service dependencies they were talking about would have been nice as well.

I mean, it's in the SLA that they have to refund 10% for the billing period IIRC.

If you request it and provide evidence that they find compelling.

To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:

the words “SLA Credit Request” in the subject line; the dates and times of each incident of non-zero Error Rates that you are claiming; and your request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks). If the Monthly Uptime Percentage applicable to the month of such request is confirmed by us and is less than the applicable Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which your request is confirmed by us. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit."

Re: Summary of the Amazon S3 Service Disruption

#349

Post by eric_h »

seanwilson wrote: post #134

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. I find making errors on production when you think you're on staging are a…

We have a deploy script that does exactly this, unfortunately we've all gotten our muscle memory so trained that most of us type the deploy command, press enter, type yes and press enter before we're ever even prompted. Fortunately in most cases a quick ctrl-c can prevent any actual damage.

Re: Summary of the Amazon S3 Service Disruption

#350

Post by canadaduane »

thraway2016 wrote: post #344

Something doesn't pass the smell test. Over two hours to reboot the index hosts?

I assume "reboot" in this instance means more than turning it off and on again--it must return to a working state, with many volumes of data requiring log processing to find the last (and best) "good state".
Post reply on HN