Live data from Hacker News

Amazon DNS error

amazon.com

51–60 of 71 posts

Re: Amazon DNS error

#51
post #41

The Route53 (AWS DNS service) is 100% SLA: http://aws.amazon.com/route53/sla/ I would hope they honor this dns issue under the same guidelines although its technically not the route53 service we are paying for.

I use Route53 with no Cloudfront involvement, and everything looks good... So I think it's not really related to Route53

Re: Amazon DNS error

#52
post #46

I'm curious if the root cause of this is software failure or external attack. Aws is clearly at the point in size and competence that hardware failures are unlikely as root cause.

> Aws is clearly at the point in size and competence that hardware failures are unlikely as root cause.

The opposite might also be true: Amazon might now have reached a point in size where they can't scale further upwards without losing visibility and control of part of their hardware.

It's all speculation anyway.

Re: Amazon DNS error

#53
post #50

[deleted]

Trust me, bare metal doesn't get you around this kind of mess, it just leads to a different set of problems. When you're core router loses both its brains because of a malfunctioning line card, all bet's are off.

Re: Amazon DNS error

#55
This is good, let's get this sort of hiccup out of the way before the predictably massive traffic from Black Friday and Cyber Monday.

What's everyone using to monitor external asset hosts? Is anyone dynamically switching between them, or failing back to local assets?

Re: Amazon DNS error

#56
post #19

Is there a RoR Gem / configuration that will serve assets locally if an external asset host name doesn't resolve or times out?

It's a decent idea, but it would be better to do client-side. For this sort of event, knowing that the name resolves on the server doesn't give you any confidence it will resolve for the client. What would be really nice is if you could specify a fallback host in your DNS prefetch, and the browser would make it "just work."

Exactly, if a POP for Australia is down but works in Germany, better to serve this on the client side.

Re: Amazon DNS error

#58

Half the internet is down, Amazon.com is flapping, the AWS website doesn't have any assets, and status.aws has a Green checkmark with a little (i), 40 minutes after the problems start?! I love AWS but they really need to improve their procedures for communicating during outages. If your parent companies' billion dollar site can be affected in any way on the night before black friday, and even your own site is down, a…

Exactly this. In our experience Amazon's status page doesn't reflects actual outages we are having, Cloudwatch often sends false positives (systems being down, when in fact, they are not) and SNS messages get lost in the ether.

We've built our systems to recover from failure states once we know there is a problem. AWS's inability to do that reliably is forcing us to own the problem ourself, and as a result, we will probably migrate away to use cheaper boxes in the cloud.

On the positive side, RDS has been solid.

Re: Amazon DNS error

#59

From what I can see myself and from twitter feed, the issue looks resolved. But the AWS status page is still not reporting any update.

"5:57 PM PST Error rates for DNS queries of CloudFront distributions are currently recovering."
Post reply on HN