Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the Amazon S3 Service Disruption

aws.amazon.com

Post reply on HN
Print view View original

451–460 of 535 posts

Re: Summary of the Amazon S3 Service Disruption

#451

Post by dsl »

chiph wrote: post #347
kevinchen wrote: post #213

Earlier quoted context omitted.

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

Or to host it on a static .html page that gets rewritten every 60 seconds or so by an external process, running on physical servers. Minimal stack, so minimal attack surface.

That is effectively what happens, they just stupidly backed that file from S3.

http://status.aws.amazon.com/data.json

Re: Summary of the Amazon S3 Service Disruption

#452

Post by patio11 »

mml wrote: post #450
patio11 wrote: post #432

Earlier quoted context omitted.

One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

Do you have a citation for that? I am curious; it's something I've never heard of and goes against my intuitions/experience regarding what traditionally managed Japanese companies would do. (Entirely possible it has happened! Hence the cite request.)

Re: Summary of the Amazon S3 Service Disruption

#453

Post by dsl »

el_benhameen wrote: post #348
CephalopodMD wrote: post #339

Earlier quoted context omitted.

I mean, it's in the SLA that they have to refund 10% for the billing period IIRC.

If you request it and provide evidence that they find compelling. To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include: the words “SLA Credit Request” in the subject line; the dates and times of each incident of non-zero Error…

> provide evidence that they find compelling

A link to their tweet about the status page not working because the building was burning down around it seems compelling.

Re: Summary of the Amazon S3 Service Disruption

#454

Post by mrep »

nsgoetz wrote: post #59

This makes me want to write a program that would ask users to confirm commands if it thinks they are running a known playbook and deviating from it. Does anyone know if a tool like that exists?

Not sure, but my company's fleet wide root scripts confirm first the exact command you want to run, then run on 1 host first and output the the full logs for you to inspect/confirm, and then finally start the full fleet wide run after you have confirmed the expected result of your output. They also output the full logs of across the entire fleet once your fleet wide script is run.

Re: Summary of the Amazon S3 Service Disruption

#455

Post by saganus »

TeMPOraL wrote: post #431
askmike wrote: post #417

Earlier quoted context omitted.

I think that even if someone can't memorize the exact characters, they'll memorize the task of having to type over the characters. Better would be to never ask for confirmation except in the worst of worst cases.

That's what I meant in my original comment when I wrote that "number of hoops you have to jump through to do something should scale with the potential impact of an operation". Harmless operations - no confirmation. Something that could mess up your work - y-or-n-p confirmation. Something that could fuck up the whole infrastructure - you'd better get ready to retype a mix of "I DO UNDERSTAND WHAT I'M JUST ABOUT TO DO"…

Not sure if even that would work.

I've almost deleted my heroku production server even though you need to type (or copy paste....ahem...) the full server name (e.g. thawing-temple-23345).

I think the reason was that because in my mind I was 100% sure this was the right server, when the confirmation came up I didn't stopped to look if indeed this was the correct one so I mechanically started to type the name of the server and just a second before I clicked ok, I had this genius idea to double check.... Oh boy... My heart dropped to the floor when I realized what was I about to do.

You could say that indeed Heroku's system of avoiding errors worked correctly....

However the confirmation dialog wasn't what made me stop... Instead it was my past-self's experience screaming at me and remembering me that ONE time where I did fucked up a production server years ago (it cost the company a full day of customers' bids... Imagine the shame of calling all the winning bidders and asking them what price did they end up bidding to win....)

My point is, maybe no number of confirmation dialogs however complex they are, will stop mistakes if the operator is fixed on doing X. If you are working in a semi-autopilot mode because you obviously are very smart and careful (ahem..) you will just do whatever the dialog asks you to do without actually thinking what you are doing.

What, then, will make you stop and verify? My only guess is that experience is the only way. I.e. only when you seriously fuck up you learn that no matter how many safety systems or complex confirmation dialogs there are you still need to double and triple check each character you typed, lest you want to go through that bad experience again....

Re: Summary of the Amazon S3 Service Disruption

#456

Post by greglindahl »

saganus wrote: post #455
TeMPOraL wrote: post #431

Earlier quoted context omitted.

That's what I meant in my original comment when I wrote that "number of hoops you have to jump through to do something should scale with the potential impact of an operation". Harmless operations - no confirmation. Something that could mess up your work - y-or-n-p confirmation. Something that could fuck up the whole infrastructure - you'd better get ready to retype a mix of "I DO UNDERSTAND WHAT I'M JUST ABOUT TO DO"…

Not sure if even that would work. I've almost deleted my heroku production server even though you need to type (or copy paste....ahem...) the full server name (e.g. thawing-temple-23345). I think the reason was that because in my mind I was 100% sure this was the right server, when the confirmation came up I didn't stopped to look if indeed this was the correct one so I mechanically started to type the name of the se…

A well-designed confirmation doesn't give you the same prompt for deleting some random test server as it does for deleting a production server. That helps with the "autopilot mode" issue.

Re: Summary of the Amazon S3 Service Disruption

#457

Post by cjs »

hibikir wrote: post #430
Bartweiss wrote: post #287

Earlier quoted context omitted.

Fair enough. I guess what I meant was specifically using training or punishment to combat "momentary lapse" issues. If someone doesn't test their backups, you train them to test backups. If someone lies about testing the backups, maybe you fire them. But if someone trips and shatters the only backup disk, you don't yell at them - you create backups that an instant of clumsiness can't ruin. I did overstate, training i…

Training about testing backups is still a bad idea: Why make someone do a job that purely verification? Those jobs eventually stop getting done, and it's hard to keep people doing them. Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those…

I agree. I have the same policy when setting up servers: don't have a "primary" and a "backup" server, make both servers production servers and have the code that uses them alternate between them, pick a random one, whatever. (I don't always get to implement this policy, of course.)

Re: Summary of the Amazon S3 Service Disruption

#458

Post by dbenhur »

evilDagmar wrote: post #439
hibikir wrote: post #430

Earlier quoted context omitted.

Training about testing backups is still a bad idea: Why make someone do a job that purely verification? Those jobs eventually stop getting done, and it's hard to keep people doing them. Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those…

No. You don't make a daily task of testing backups. That would be wrong for precisely the reasons you cite. It's a waste of effort and time, and ignores what the point of testing them is for: ensuring that the procedure still works . One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. The rest of the year it's only worth keeping an automated pr…

> One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data.

Nope. Nope. Nope.

You test every backup by automatically restoring from it in a sandbox and verifying its integrity and functionality in the restored state.

Backups are worthless unless verified for their intended use of recovering a functioning system.

Re: Summary of the Amazon S3 Service Disruption

#459

Post by saganus »

greglindahl wrote: post #456
saganus wrote: post #455

Earlier quoted context omitted.

Not sure if even that would work. I've almost deleted my heroku production server even though you need to type (or copy paste....ahem...) the full server name (e.g. thawing-temple-23345). I think the reason was that because in my mind I was 100% sure this was the right server, when the confirmation came up I didn't stopped to look if indeed this was the correct one so I mechanically started to type the name of the se…

A well-designed confirmation doesn't give you the same prompt for deleting some random test server as it does for deleting a production server. That helps with the "autopilot mode" issue.

I agree that it should help reduce the amount of mistakes.

But I still believe auto-pilot mode is a real thing (and a danger!) .

My point is that I'm not sure if it's even possible to design one that actually cuts errors to 0.

And if that's indeed the case, even if it's close to 0, it's still non-zero, thus at the scale Amazon operates at, it's very probable that it will happen at least one time.

Maybe sometime in the future AI systems will help here?

Re: Summary of the Amazon S3 Service Disruption

#460

Post by brianpan »

chrisseaton wrote: post #189
brianpan wrote: post #182

Earlier quoted context omitted.

A good UI should be able to help, especially in critical situations. I imagine Amazon will consider something like this: > The dosage you ordered is an order of magnitude greater than the dosage most commonly ordered for this medicine. Continue? y/n

But that still allows you to make a mistake - by pressing y when that's the wrong thing to do.

This prevents fat-finger mistakes.

You could go further and try to prevent cat-on-the-keyboard mistakes, which is maybe what you're describing (solve this math equation to prove you are a human who is sufficiently not inebriated). Or even further and prevent malicious, trench-coat wearing, pointy-nosed trouble-makers.

The point is, yes, it is possible. That's what good design does.

Post reply on HN