Akamai Edge DNS was down • Hacker News Board

Re: Akamai Edge DNS was down

#51

Post by grishka » Thu, Jul 22, 2021, 4:35 PM UTC

davidjgraph wrote: post #15

Serious question, has anyone properly solved the issue of DNS as a single point of failure?

And then there are Cloudflare and other Centralized Downtime Networks as another point of failure.

Re: Akamai Edge DNS was down

#52

Post by SjorsVG » Thu, Jul 22, 2021, 4:35 PM UTC

Many banks in the Netherlands are affected by this.

Re: Akamai Edge DNS was down

#53

Post by tootie » Thu, Jul 22, 2021, 4:35 PM UTC

MrRadar wrote: post #21

tru3_power wrote: post #16

Any idea on cause? Ddos or hardware failure?

Widespread issues like this on major CDNs tend to be configuration errors.

Cloudflare seems to be struggling too. Not sure if they have some dependency on Akamai or if this portends something much worse

Re: Akamai Edge DNS was down

#54

Post by schemathings » Thu, Jul 22, 2021, 4:35 PM UTC

Possibly related .. Verizon peering issues / ASN701 at Equinix NY2 in Secaucus NJ

Re: Akamai Edge DNS was down

#55

Post by knaik94 » Thu, Jul 22, 2021, 4:36 PM UTC

I am surprised financial institutions don't have any regulation for redundancy. The one that stuck out to me is the Navy Federal Credit Union website being down. I have not had any issues logging into mobile though for some of the reported sites.

Re: Akamai Edge DNS was down

#56

Post by rvz » Thu, Jul 22, 2021, 4:36 PM UTC

whitepoplar wrote: post #23

rvz wrote: post #12

Probably Akamai needs to use Kubernetes. EDIT: So HN can't even take a joke after this? [0] [0] https://news.ycombinator.com/item?id=27893482

Probably caused by Kubernetes

That's even worse if true; despite HNers creating a storm in a tea cup on DOSing a blog of a service not using K8s when having a blog is not their main service. [0].

Either way, the joke's is now on the HNers in that thread.

[0] https://news.ycombinator.com/item?id=27893482

Re: Akamai Edge DNS was down

#57

Post by compscistd » Thu, Jul 22, 2021, 4:36 PM UTC

nonfamous wrote: post #42

delgaudm wrote: post #22

Lastpass is down, so if you use lastpass the effect is significantly compounded.

It still works in offline mode. You can’t update passwords, but you can retrieve them.

To enable offline mode, I had to turn on airplane mode on my phone before logging in.

Re: Akamai Edge DNS was down

#58

Post by tjpnz » Thu, Jul 22, 2021, 4:37 PM UTC

Just got booted out of Netflix on the PS4 because the console could no longer connect to Sony's license server. Netflix was working just fine by the way.

Re: Akamai Edge DNS was down

#59

Post by sathackr » Thu, Jul 22, 2021, 4:37 PM UTC

cbono1 wrote: post #38

Why would Google and Amazon be on the downdetector list or experiencing issues? Don't they have their own DNS / nameservers separate from Akamai?

because the way downdetector works is it just basically counts how many people are searching/visiting for down and if it's much higher than typical it flags the site as down.

So if everyone searched "is google down" and visited the link on downdetector that was returned in the search, that would add to the downdetector count for that site.

Downdetector doesn't actually know if the site is up or down.

Re: Akamai Edge DNS was down

#60

Post by uncertainrhymes » Thu, Jul 22, 2021, 4:37 PM UTC

mvanaltvorst wrote: post #39

What role does Akamai Edge DNS play in normal internet traffic? DNS responses usually get cached, as far as I understand correctly. And it is usually possible to change your DNS server to e.g. Google's and circumvent the outage. Does Akamai Edge DNS play a role on the server side?

If you use a CDN to front your traffic, you need the CNAME for www (or whatever) to be pointing at their DNS infrastructure, so they can return whichever closest POP is going to serve your traffic.

e.g. dig @1.1.1.1 www.nvidia.com +trace

... various things from the root ...

www.nvidia.com. 7200 IN CNAME www.nvidia.com.edgekey.net. ;; Received 83 bytes from 208.94.148.13#53(ns5.dnsmadeeasy.com) in 35 ms

So the main DNS is fine, but it'll never get an A record because the last link in the chain is toast -- edgekey being Akamai in this case, but all CDNs do this so they can route traffic. Normally, this is a good thing so they can shift traffic within 30 seconds on their side. Unfortunately, it also means it would take nvidia an two hours to point away from Akamai.