Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon DNS error

amazon.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Amazon DNS error

#62

Post by atom_enger »

anonymuse wrote: post #55

This is good, let's get this sort of hiccup out of the way before the predictably massive traffic from Black Friday and Cyber Monday. What's everyone using to monitor external asset hosts? Is anyone dynamically switching between them, or failing back to local assets?

This was linked elsewhere in the thread https://github.com/etsy/cdncontrol/ and Etsy seemed to be up through the entire thing. So perhaps they're doing something right :)

Re: Amazon DNS error

#63

Post by fletchowns »

jdotjdot wrote: post #47
jordanthoms wrote: post #35

Earlier quoted context omitted.

We have business-level support through AWS activate, and it's not any different really - we could submit a ticket and get a phone call back within a 1 hour guarantee, but I'm not sure that would get you much more information or faster resolution for an issue like this. The really big customers presumably have contacts within the AWS team who give them extra information in cases like this.

I had an Urgent-level issue just two days ago (1-hr SLA), also with business-level support through AWS Activate, and it took them THREE hours to get back to me, with me simultaneously waiting on one chat and two phone lines to try to get in contact with anyone I could, Finally, chat came through--but only after a wait far longer than what their SLA guarantees. Point being that support probably isn't going to get you…

So what's the consequence of them violating the SLA?

Re: Amazon DNS error

#64

Post by ams6110 »

rdl wrote: post #46

I'm curious if the root cause of this is software failure or external attack. Aws is clearly at the point in size and competence that hardware failures are unlikely as root cause.

But aren't big, complex systems more likely to fail in unexpected ways?

Re: Amazon DNS error

#65

Post by Hengjie »

6:24 PM PST Between 4:12 PM and 6:02 PM PST, users experienced elevated error rates when making DNS queries for CloudFront distributions. The service has recovered and is operating normally.

Re: Amazon DNS error

#66

Post by [deleted] »

erichmond wrote: post #58
jordanthoms wrote: post #27

Half the internet is down, Amazon.com is flapping, the AWS website doesn't have any assets, and status.aws has a Green checkmark with a little (i), 40 minutes after the problems start?! I love AWS but they really need to improve their procedures for communicating during outages. If your parent companies' billion dollar site can be affected in any way on the night before black friday, and even your own site is down, a…

Exactly this. In our experience Amazon's status page doesn't reflects actual outages we are having, Cloudwatch often sends false positives (systems being down, when in fact, they are not) and SNS messages get lost in the ether. We've built our systems to recover from failure states once we know there is a problem. AWS's inability to do that reliably is forcing us to own the problem ourself, and as a result, we will p…

[deleted]

Re: Amazon DNS error

#67

Post by jacobsenscott »

cjreyes wrote: post #19

Is there a RoR Gem / configuration that will serve assets locally if an external asset host name doesn't resolve or times out?

You can assign a proc to config.asset_host, so you could easily do it -

  config.asset_host = -> {
    cdn_up? ? "http://mycdn.com" : "http://mydomain.com"
  }

Re: Amazon DNS error

#68

Post by felixgallo »

jordanthoms wrote: post #27

Half the internet is down, Amazon.com is flapping, the AWS website doesn't have any assets, and status.aws has a Green checkmark with a little (i), 40 minutes after the problems start?! I love AWS but they really need to improve their procedures for communicating during outages. If your parent companies' billion dollar site can be affected in any way on the night before black friday, and even your own site is down, a…

Imagine what kind of conditions would have to exist for Amazon to use a red indicator.

Re: Amazon DNS error

#69

Post by stevekemp »

ejdyksen wrote: post #14

Amazon appears to be up for me, but a number of places are still affected: - All of Vox Media's properties (The Verge, Polygon, Vox.com, SBNation, etc) - All of Atlasssian's services (Bitbucket, Jira OnDemand, etc) - Flowdock - Instagram - aws.amazon.com has no assets Edit: console.aws.amazon.com has no assets, either, so it's also currently worthless. It's probably worth having a DNS failover strategy for Route53 (i…

> It's probably worth having a DNS failover strategy for Route53 (if that's what you're using) that doesn't involve the UI on console.aws.amazon.com.

Which is one of the reasons I setup https://dns-api.com/ - A way of updating Route53 DNS via git hooks.

Post reply on HN