Amazon DNS error • Hacker News Board

Re: Amazon DNS error

#21

Post by dice » Thu, Nov 27, 2014, 1:01 AM UTC

WestCoastJustin wrote: post #2

UPDATE: RESOLVED according to the AWS status page -- 6:24 PM PST Between 4:12 PM and 6:02 PM PST, users experienced elevated error rates when making DNS queries for CloudFront distributions. The service has recovered and is operating normally. [2] -- Cloudfront DNS is hosed. Doing a DNS lookup on my cloudfront distribution fails. Many folks on Twitter also see the issue too [1]. Maybe a failed upgrade or something, t…

The AWS status page not showing anything isn't unusual. They'll probably update in half an hour to describe it as a partial failure then revise it to an "all OK" 10 minutes before it's actually fixed then retroactively downgrade it to a minor quality of service disruption.

Not that I've noticed they tend to lie through their teeth on the status page or anything....

Re: Amazon DNS error

#22

Post by zuccs » Thu, Nov 27, 2014, 1:02 AM UTC

stephenc_c_ wrote: post #9

My site is down, I'm using Route53, S3 and CloudFront so I guess that's why.

Same setup here except Cloudflare instead of Route53. So I guess it's S3/CloudFront related?

Re: Amazon DNS error

#23

Post by ranman » Thu, Nov 27, 2014, 1:03 AM UTC

Status page is updated: http://status.aws.amazon.com/

Re: Amazon DNS error

#24

Post by bbnnt » Thu, Nov 27, 2014, 1:04 AM UTC

cjreyes wrote: post #19

Is there a RoR Gem / configuration that will serve assets locally if an external asset host name doesn't resolve or times out?

that'd be genius

Re: Amazon DNS error

#25

Post by msaffitz » Thu, Nov 27, 2014, 1:06 AM UTC

ranman wrote: post #23

Status page is updated: http://status.aws.amazon.com/

We were first alerted to an issue by our monitoring at 4:20 Pacific, 40 minutes before AWS updated their status page. It's infuriating that AWS doesn't communicate more promptly and transparently when an issues like this occurs.

Re: Amazon DNS error

#26

Post by teraflop » Thu, Nov 27, 2014, 1:07 AM UTC

dice wrote: post #21

WestCoastJustin wrote: post #2

UPDATE: RESOLVED according to the AWS status page -- 6:24 PM PST Between 4:12 PM and 6:02 PM PST, users experienced elevated error rates when making DNS queries for CloudFront distributions. The service has recovered and is operating normally. [2] -- Cloudfront DNS is hosed. Doing a DNS lookup on my cloudfront distribution fails. Many folks on Twitter also see the issue too [1]. Maybe a failed upgrade or something, t…

The AWS status page not showing anything isn't unusual. They'll probably update in half an hour to describe it as a partial failure then revise it to an "all OK" 10 minutes before it's actually fixed then retroactively downgrade it to a minor quality of service disruption. Not that I've noticed they tend to lie through their teeth on the status page or anything....

Apparently, "DNS resolution errors" are considered neither a "service disruption" nor a "performance issue", but instead are categorized as an "informational message."

Re: Amazon DNS error

#27

Post by jordanthoms » Thu, Nov 27, 2014, 1:10 AM UTC

Half the internet is down, Amazon.com is flapping, the AWS website doesn't have any assets, and status.aws has a Green checkmark with a little (i), 40 minutes after the problems start?!

I love AWS but they really need to improve their procedures for communicating during outages. If your parent companies' billion dollar site can be affected in any way on the night before black friday, and even your own site is down, and you are not acknowledging that the service is FUBAR - you have a problem.

Re: Amazon DNS error

#28

Post by jbinto » Thu, Nov 27, 2014, 1:12 AM UTC

ejdyksen wrote: post #14

Amazon appears to be up for me, but a number of places are still affected: - All of Vox Media's properties (The Verge, Polygon, Vox.com, SBNation, etc) - All of Atlasssian's services (Bitbucket, Jira OnDemand, etc) - Flowdock - Instagram - aws.amazon.com has no assets Edit: console.aws.amazon.com has no assets, either, so it's also currently worthless. It's probably worth having a DNS failover strategy for Route53 (i…

Amazon.com is up for me, but is entirely missing product images.

Re: Amazon DNS error

#29

Post by pauloschilling » Thu, Nov 27, 2014, 1:12 AM UTC

msaffitz wrote: post #25

ranman wrote: post #23

Status page is updated: http://status.aws.amazon.com/

We were first alerted to an issue by our monitoring at 4:20 Pacific, 40 minutes before AWS updated their status page. It's infuriating that AWS doesn't communicate more promptly and transparently when an issues like this occurs.

+1

Re: Amazon DNS error

#30

Post by fletchowns » Thu, Nov 27, 2014, 1:13 AM UTC

msaffitz wrote: post #25

ranman wrote: post #23

Status page is updated: http://status.aws.amazon.com/

We were first alerted to an issue by our monitoring at 4:20 Pacific, 40 minutes before AWS updated their status page. It's infuriating that AWS doesn't communicate more promptly and transparently when an issues like this occurs.

Is it any different if you shell out the big bucks for support?