Live data from Hacker News

Akamai Edge DNS was down

edgedns.status.akamai.com

71–80 of 229 posts

Re: Akamai Edge DNS was down

#71
post #55

I am surprised financial institutions don't have any regulation for redundancy. The one that stuck out to me is the Navy Federal Credit Union website being down. I have not had any issues logging into mobile though for some of the reported sites.

CapitalOne has a broken login which is pretty surprising to me.

Re: Akamai Edge DNS was down

#73
post #7

You can see this on a lot of sites right now. You get the Akamai style error with something like: Reference: #11.453a2f17.1393u44848484.3aee33433 At the bottom of a very bland looking error page.

What's frustrating is that DNS is returning an address, instead of just failing, and so macos is caching that value (though it might be cloudflare doing that).

Re: Akamai Edge DNS was down

#76

Serious question, has anyone properly solved the issue of DNS as a single point of failure?

It is relatively easy to make DNS highly redundant: just put multiple DNS server in data-centers which are as independent as possible (different geo locations, different ISPs). You can also use different DNS software and different OS (say BSD+Linix) to exclude correlated bugs. Root DNS server AFAIK use different software for this reason.

Problems starts when you want to easy make frequent changes and introduce complex software to manage DNS zones (and complexity usually comes with bugs).

Re: Akamai Edge DNS was down

#77
post #49

Serious question, has anyone properly solved the issue of DNS as a single point of failure?

Depending on what point you draw the line of "single point of failure" you could use multiple providers for your dns. GOV.UK for example uses both aws and gcp for DNS

So, NS entries pointing to both? But then take the example your domain was in Route53 and AWS goes down. You can't configure the NS entries to avoid AWS DNS servers. Is the idea that child DNS servers detect the outage and cache the values in the name server(s) that remain up?

But then, the cached values from AWS take a while to clear, TTL never seems to be applied properly. It always feels like the worst case in such a scenario is you can point everyone at the right thing within 24 hours.

Re: Akamai Edge DNS was down

#79
post #38

Why would Google and Amazon be on the downdetector list or experiencing issues? Don't they have their own DNS / nameservers separate from Akamai?

because the way downdetector works is it just basically counts how many people are searching/visiting for down and if it's much higher than typical it flags the site as down. So if everyone searched "is google down" and visited the link on downdetector that was returned in the search, that would add to the downdetector count for that site. Downdetector doesn't actually know if the site is up or down.

A more proper name might be PeopleThinkItsDownDetector.com

Re: Akamai Edge DNS was down

#80

What role does Akamai Edge DNS play in normal internet traffic? DNS responses usually get cached, as far as I understand correctly. And it is usually possible to change your DNS server to e.g. Google's and circumvent the outage. Does Akamai Edge DNS play a role on the server side?

Looks like this: the affected subdomains are CNAMEd to the akamai CDN, and the Nameserver for those are/were down.

So for example:

Top level domain for nvidia resolved fine..

dig @1.1.1.1 nvidia.com => status: NOERROR, Nameservers are ns6.dnsmadeeasy.com

But the website didnt. dig @1.1.1.1 www.nvidia.com => status: SERVFAIL,

The Nameserver for the this www.nvidia resolved to the akamai nameserver which had a problem..

dig @1.1.1.1 www.nvidia.com NS => CNAME e33907.a.akamaiedge.net.

Post reply on HN