AWS Route53 is down

#1

Post by paps » Tue, Apr 24, 2018, 12:37 PM UTC

AWS Route53 is down
twitter.com

Re: AWS Route53 is down

#2

Post by scarface74 » Tue, Apr 24, 2018, 1:53 PM UTC

Route 53 I believe is the only service that AWS promises "100% uptime". So much for that....

Re: AWS Route53 is down

#3

Post by evfanknitram » Tue, Apr 24, 2018, 2:04 PM UTC

scarface74 wrote: post #2

Route 53 I believe is the only service that AWS promises "100% uptime". So much for that....

The Twitter post says it's Google DNS having issues querying. Are you sure the issue is with AWS Route 53 and not some routing issue between Google and AWS?

Re: AWS Route53 is down

#4

Post by loriverkutya » Tue, Apr 24, 2018, 2:12 PM UTC

"works on my machine"

Our infrastructure is heavily using Route53 and we are not seeing any outage. (~1000 domains)

Edit: from http://status.aws.amazon.com/

"5:19 AM PDT We are investigating reports of problems resolving some DNS records hosted on Route53 using the third party DNS resolvers 8.8.8.8 and 8.8.4.4 . DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers are not affected at this time.

5:49 AM PDT We have identified the cause for an elevation in DNS resolution errors using third party DNS resolvers 8.8.8.8 / 8.8.4.4 and are working towards resolution. DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers continues to work normally.

6:10 AM PDT Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced elevated errors resolving DNS records hosted on Route 53 using DNS resolvers 8.8.8.8 / 8.8.4.4 . The issue has been resolved and the service is operating normally."

Re: AWS Route53 is down

#5

Post by lightbyte » Tue, Apr 24, 2018, 3:28 PM UTC

scarface74 wrote: post #2

Route 53 I believe is the only service that AWS promises "100% uptime". So much for that....

Is it actually possible for any system to have 100% uptime? Or do they just use that as a marketing gimmick and give out credit whenever there is downtime.

Re: AWS Route53 is down

#6

Post by masukomi » Tue, Apr 24, 2018, 3:53 PM UTC

scarface74 wrote: post #2

Route 53 I believe is the only service that AWS promises "100% uptime". So much for that....

Some people are just having dns issues getting there. It's like saying my house is still intact even if you lost your map to it. You loosing the map does not impact the question of if my house is still standing or not.

Re: AWS Route53 is down

#7

Post by masukomi » Tue, Apr 24, 2018, 3:55 PM UTC

lightbyte wrote: post #5

scarface74 wrote: post #2

Route 53 I believe is the only service that AWS promises "100% uptime". So much for that....

Is it actually possible for any system to have 100% uptime? Or do they just use that as a marketing gimmick and give out credit whenever there is downtime.

sure, why not? I mean, assuming there isn't some nuclear attack that destroys all networking infrastructure. I'm not suggesting it's easy, but if you distribute it enough and make it able to work when other portions of the system disappear i see no reason you couldn't claim 100% uptime

Re: AWS Route53 is down

#8

Post by omarforgotpwd » Tue, Apr 24, 2018, 4:08 PM UTC

This was probably just due to someone being stupid but the conspiracy theorist in me asks: Sabotage by the Google Cloud team?

Re: AWS Route53 is down

#9

Post by jbergstroem » Tue, Apr 24, 2018, 4:25 PM UTC

I saw issues resolving via azure dns servers (i have gitlab runners on there) as well:

ssh 10.0.1.7 ping -c 1 github-production-release-asset-2e65be.s3.amazonaws.com

ping: github-production-release-asset-2e65be.s3.amazonaws.com: Temporary failure in name resolution```

Re: AWS Route53 is down

#10

Post by evfanknitram » Tue, Apr 24, 2018, 4:35 PM UTC

masukomi wrote: post #7

lightbyte wrote: post #5

Earlier quoted context omitted.

Is it actually possible for any system to have 100% uptime? Or do they just use that as a marketing gimmick and give out credit whenever there is downtime.

sure, why not? I mean, assuming there isn't some nuclear attack that destroys all networking infrastructure. I'm not suggesting it's easy, but if you distribute it enough and make it able to work when other portions of the system disappear i see no reason you couldn't claim 100% uptime

And then you you have a stupid software bug and the redundant physical systems are not helping.

Azure Storage is a available in a ton of locations and all of them stopped working because Microsoft forgot to roll out their patch to renew the SSL certificate. So they knew about the issue but rolling out the patch was not done.

Another example was when they introduced HTTP2 for CDN endpoints and forgot to test the cipher list with popular browsers and broke compatibility with Firefox ESR.