Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hello admins of EU AWS systems – DNS outage in eu-west-1

news.ycombinator.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#11

Post by aidos »

danielhunt wrote: post #9
aidos wrote: post #7

Earlier quoted context omitted.

I recall reading somewhere many years ago that AWS randomise a, b, c etc per account. Otherwise people have a tendency to pick A and be done with it.

That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

I don't think I explained that very clearly.

What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones:

    zone X (real) : your A (virtual) : my C (virtual)
    zone Y (real) : your B (virtual) : my B (virtual)
    zone Z (real) : your C (virtual) : my A (virtual)
That's why people see different outage characteristics between the zones.

Edit: one of the few articles (other than HN comments) that explains it https://alestic.com/2009/07/ec2-availability-zones/

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#12

Post by danielhunt »

aidos wrote: post #11
danielhunt wrote: post #9

Earlier quoted context omitted.

That's true - they loadbalance instance creation unless you specifically ask for an instance in a zone That doesn't apply here though. Instances that are already up are experiencing DNS resolution issues because the Amazon-provided DNS service (which is distinct from Route53) is failing

I don't think I explained that very clearly. What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones: zone X (real) : your A (virtual) : my C (virtual) zone Y (real) : your B (virtual) : my B (virtual) zone Z (real) : your C (virtual) : my A (virtual) That's why people see different outage characteristics between the zones. E…

I've been using AWS for years and this is the first I've ever heard of this.

All I can say is "WHAAAAAAT?!"

You've made me question my reality. Take an upvote for that alone.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#13

Post by aidos »

danielhunt wrote: post #12
aidos wrote: post #11

Earlier quoted context omitted.

I don't think I explained that very clearly. What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones: zone X (real) : your A (virtual) : my C (virtual) zone Y (real) : your B (virtual) : my B (virtual) zone Z (real) : your C (virtual) : my A (virtual) That's why people see different outage characteristics between the zones. E…

I've been using AWS for years and this is the first I've ever heard of this. All I can say is "WHAAAAAAT?!" You've made me question my reality. Take an upvote for that alone.

I'm glad I could make what is otherwise a bad time to be looking after servers a little more enjoyable.

I myself was in the middle of a release when all this happened. "I'll just release this thing" I said, "will only take 20 minutes" I said....

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#14

Post by jlgaddis »

aidos wrote: post #8

Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves? When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

I'm not a big user of AWS but I don't know that there's anything stopping you from running your own recursive resolver (in your VPC) that your other machines use (instead of using an external resolver or whatever Amazon provides).

You could, for instance, stand up an instance or two running Unbound, forward to Amazon's own resolvers for internal stuff, and forward to external resolvers for external resolution (or just handle it yourself instead of using forwarders).

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#15

Post by tomekit »

jlgaddis wrote: post #14
aidos wrote: post #8

Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves? When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

I'm not a big user of AWS but I don't know that there's anything stopping you from running your own recursive resolver (in your VPC) that your other machines use (instead of using an external resolver or whatever Amazon provides). You could, for instance, stand up an instance or two running Unbound, forward to Amazon's own resolvers for internal stuff, and forward to external resolvers for external resolution (or jus…

The main reason to use AWS is so you don't need to do anything except calculate profit from your app, at least in theory. If you need to set up own resolvers or anything else within the AWS there is no reason to use AWS in the first place.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#16

Post by chatmasta »

aidos wrote: post #8

Question for the more experienced in the group - is there a way of caching the DNS locally within you VPC in case something like this happens again? Or does that just cause more issues than it solves? When I discovered it I tried switching my resolv.conf to 8.8.8.8 but of course none of my internal stuff worked because of how my security group / subnet / ip restrictions are setup internally.

Yes, you can specify DNS server to use in the DHCP options set of the VPC. I've done this to point to internal DNS servers.

Re: Hello admins of EU AWS systems – DNS outage in eu-west-1

#17

Post by misframer »

danielhunt wrote: post #12
aidos wrote: post #11

Earlier quoted context omitted.

I don't think I explained that very clearly. What you see as zone A is not what I see as zone A (maybe). When you sign up for an account, AWS assigns your zone A to one of the 3 available zones: zone X (real) : your A (virtual) : my C (virtual) zone Y (real) : your B (virtual) : my B (virtual) zone Z (real) : your C (virtual) : my A (virtual) That's why people see different outage characteristics between the zones. E…

I've been using AWS for years and this is the first I've ever heard of this. All I can say is "WHAAAAAAT?!" You've made me question my reality. Take an upvote for that alone.

It's in the documentation [0].

    An Availability Zone is represented by a region code followed by a letter identifier;
    for example, us-east-1a. To ensure that resources are distributed across the Availability
    Zones for a region, we independently map Availability Zones to identifiers for each
    account. For example, your Availability Zone us-east-1a might not be the same location
    as us-east-1a for another account. There's no way for you to coordinate Availability
    Zones between accounts.
[0] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
Post reply on HN