Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoDaddy outage caused by corrupted router tables

godaddy.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: GoDaddy outage caused by corrupted router tables

#21

Post by nettdata »

overworkedasian wrote: post #11

I wasnt really paying attention to the outage, but if it was indeed a routing issue, then you shouldnt have been able to reach any godaddy ip address. ICMP/traceroutes would have failed and showed the error.

Why would you say that? Large networks have a ton of routers in them, and even a lot of switches provide routing functionality.

I don't know the details of the environment, but even in smaller systems I've worked on there is a fair bit of hardware separation between various network segments. Complete failure on one part would not affect the others.

For that matter, even a slight corruption in some ARP caches, or stale internal tables, etc., could cause the problems they had... it's not just a complete failure that could cause problems.

And "routing" is such a generic term, when it could really be any number of feature sets that failed; load balancing, source routing configs, etc.

Re: GoDaddy outage caused by corrupted router tables

#23

Post by highfreq »

How can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.

Re: GoDaddy outage caused by corrupted router tables

#24

Post by mikedouglas »

highfreq wrote: post #23

How can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.

I think what they're trying to say is that 99% of the time, they have 99.999% uptime.

Re: GoDaddy outage caused by corrupted router tables

#25

Post by JoeAltmaier »

mikedouglas wrote: post #24
highfreq wrote: post #23

How can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.

I think what they're trying to say is that 99% of the time, they have 99.999% uptime.

Silly: 99% of the time they have 100% uptime.

Re: GoDaddy outage caused by corrupted router tables

#26

Post by lamebrain »

highfreq wrote: post #23

How can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.

"99.999% starting... now."

Re: GoDaddy outage caused by corrupted router tables

#27

Post by nettdata »

druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Unless you have intimate knowledge of their network topology, and know the specifics of where those pinged IP's live in that topology, and what routes were used to provide DNS results, you can't say that it wasn't a routing issue.

"Routing" is a rather generic term when it comes to large networks, and everything from border routers, firewalls, load balancers, and switches actually perform routing.

Especially (as I've mentioned in another post) when you add fault tolerance / failover configurations to the mix.

Routing failure doesn't have to be an all-or-nothing thing. There are a number of ways in which I can see ICMP echo packets working but other traffic not, especially when you include complexities of source routing, load balancing, failover, etc.

Even something as "simple" as a poisoned ARP cache in a single box could screw up the entire internal network and cause the problems they've had, and still be considered a "routing issue".

$0.02

Re: GoDaddy outage caused by corrupted router tables

#28

Post by mootothemax »

I have to wonder how many extra customers the various third-party DNS services have gained as a direct result of this.

I've just switched to DNSMadeEasy - for anyone concerned about the time involved, they have some cool timesavers like templates you can apply to all of your domains at once. Really makes a difference not having to manually set up individually the entries for Google Apps on 20+ domains.

Re: GoDaddy outage caused by corrupted router tables

#29

Post by druiid »

nettdata wrote: post #27
druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Unless you have intimate knowledge of their network topology, and know the specifics of where those pinged IP's live in that topology, and what routes were used to provide DNS results, you can't say that it wasn't a routing issue. "Routing" is a rather generic term when it comes to large networks, and everything from border routers, firewalls, load balancers, and switches actually perform routing. Especially (as I've…

None of that is necessarily incorrect... but per their news release 'corrupted router data tables' (their words) were the issue. I can't read too much into that, but that still doesn't change the fact that DNS wasn't resolving for a while after they made their Verisign change (for clients), yet their website was resolved with this change.

You are correct that I don't know the details of their internal network and I never said otherwise, just that the chain of events and their claims don't necessarily match up!

Post reply on HN