Hello admins of EU AWS systems – DNS outage in eu-west-1 • Hacker News Board

Hello admins of EU AWS systems – DNS outage in eu-west-1

#1

Post by danielhunt » Thu, Dec 15, 2016, 2:23 AM UTC

https://status.aws.amazon.com

Looks like there's a DNS outage in AWS Dublin

All DNS request coming from `eu-west-1b` fail completely if you're using the default AWS DNS resolution itself - if you switch to `8.8.8.8` it's all totally fine, but that's of no use if you have private services that need internal AWS resolution

`1a` and `1c` are also affected, but to a much lower extent from what I can tell

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#2

Post by neo2001 » Thu, Dec 15, 2016, 2:31 AM UTC

sigh

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#3

Post by paugay » Thu, Dec 15, 2016, 2:32 AM UTC

same here :) let's see how long it takes

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#4

Post by neo2001 » Thu, Dec 15, 2016, 2:32 AM UTC

6:29 PM PST We have identified the root cause of the DNS resolution issues in the EU-WEST-1 Region and continue working towards resolution.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#5

Post by fern4lvarez » Thu, Dec 15, 2016, 2:37 AM UTC

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#6

Post by danielhunt » Thu, Dec 15, 2016, 2:44 AM UTC

fern4lvarez wrote: post #5

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

Interesting difference in experience there

Complicated networks are complicated, I suppose

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#7

Post by aidos » Thu, Dec 15, 2016, 2:46 AM UTC

fern4lvarez wrote: post #5

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#8

Post by aidos » Thu, Dec 15, 2016, 2:49 AM UTC

Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves?

When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#9

Post by danielhunt » Thu, Dec 15, 2016, 2:55 AM UTC

aidos wrote: post #7

fern4lvarez wrote: post #5

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone

That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#10

Post by danielhunt » Thu, Dec 15, 2016, 2:55 AM UTC

fern4lvarez wrote: post #5

It's weird, I see actually most of my issues coming from instances located in `eu-west-1a`, whereas `eu-west-1b` and `eu-west-1c` look fine.

Seeing an increase in errors in `1a` and `1c` now myself too