I wasnt really paying attention to the outage, but if it was indeed a routing issue, then you shouldnt have been able to reach any godaddy ip address. ICMP/traceroutes would have failed and showed the error.
GoDaddy outage caused by corrupted router tables
11–20 of 97 posts
Re: GoDaddy outage caused by corrupted router tables
#12"The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack." So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? The router tables just corrupted themselves?
> So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? Yeah, with a name/handle like Anonymous0wn3r they sound very trustworthy. If they claimed they had a hand in something, it must be true.
I don't trust Anonymous0wn3r but I distrust GoDaddy just the same. Of course those aren't the only two answers, it could have been external and not Anonymous0wn3r as well.
Re: GoDaddy outage caused by corrupted router tables
#13"The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack." So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? The router tables just corrupted themselves?
> So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in?
Gee, why would an anonymous person on the internet lie?
Re: GoDaddy outage caused by corrupted router tables
#14I don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?
Routers at this level aren't just scale-ups of your home wifi/nat box. They aren't even scale ups of the simple IP routers for a basic IT data-closet that manages subnets and whatnot (already much more complex by dealing with vlans and subnets and dmz and vpn issues). At the level of big networking company they are a truly complex beast.
Just at the IP level they have to deal with (at the edges and across substantial WANS) BGP - a notoriously ugly and fragile protocol. Internal routing protocols such a OSPF are equally ugly and prone to breakage. Many are the tales of some small company misconfiguring their edge routers slightly (say a 1 char typo) and having the entire internet route through their T1, across their lan, out their backup T1. Other issues are BGP flapping, resulting in scary percentages of lost traffic. This doesn't even cover trickier stuff like routing loops...
Other considerations in big routers are things like ASN identifiers and peering points. Considerations like traffic cost, SLAs and QoS all go in to traffic balancing on such routers. MPLS clouds complicate (and oddly enough simplify) these things as well.
There are also important issues like Anycast, CDN and NAT that largely rely on router tricks and add to the complication.
Finally, on top of all this, is the security concern - you can't just throw a firewall in front of it, as many firewall issues are routing issues, therefore must also be present in the router.
All these layers interact and affect each other. Any given machine can only handle so much traffic and so many decisions, so something that is drawn as a single router on a networking chart may actually be several boxes cascaded to handle the complexity.
Oh yeah, and switches are getting progressively smarter with other rules and weirdnesses that provide horribly leaky abstractions that shouldn't matter to the upstream router, but turn out to add issues to the configuration and overall complexity on top of it all.
Re: GoDaddy outage caused by corrupted router tables
#15I don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?
Routers can get quite complicated. At a first level, routers will cache information about what machines are on which port of the router. This enables them to not forward a packet to every port on the router (reducing network load). This is normally done using MAC Addresses. On more expensive routers, the router can understand IGMP, or multicast and route based on multicast joins. This enables an optimization on simpl…
Never mind when you start adding fault-tolerance and failover setups/configs to the mix.
It gets really complex really fast, and even the slightest lapse in attention to extreme detail can be disastrous and a nightmare to troubleshoot.
For instance, I once had a router reboot for some unknown reason, and the firmware/config wasn't properly flashed, so it reverted to the last point release of the firmware. That was enough to cause the failover heartbeat to be constantly triggered, and the master/slave routers just kept failing over to each other and corrupted all the ARP caches. Makes for a lot of fun when things work and then stop working in an inconsistent manner.
Re: GoDaddy outage caused by corrupted router tables
#16The DNS is designed to provide resiliency to these kinds of problems by providing the ability to list multiple NS records located in different networks. It is standard practice for top-level domain operators and other high-activity domains to place their name servers in different networks to guard against these kinds of issues. When companies put all their name servers in the same network, they are removing the diversity benefit and create a single point of failure. Domain operators should take this as a cautionary tale that they shouldn't have all their eggs in one basket and make sure a single network failure couldn't take all their name servers offline.
Re: GoDaddy outage caused by corrupted router tables
#17I don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?
They're not that simple due to aggressive speed optimizations like caching frequently used routing information in hardware registers. If the cache gets corrupted due to electronic bogon flux, bad things happen. These events are rare (like once every billion operating hours) so you can imagine it's hard for the router vendors to find & fix them all. It is a little sad that they didn't try rebooting the routers for so…
> It is a little sad that they didn't try rebooting the routers for so many hours.
This assumes they're being literal and "corrupt" means "a bit randomly flipped" instead of using "corrupt" in the figurative sense of "operator error".
Re: GoDaddy outage caused by corrupted router tables
#18I don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?
Routers at this level aren't just scale-ups of your home wifi/nat box. They aren't even scale ups of the simple IP routers for a basic IT data-closet that manages subnets and whatnot (already much more complex by dealing with vlans and subnets and dmz and vpn issues). At the level of big networking company they are a truly complex beast. Just at the IP level they have to deal with (at the edges and across substantial…
...and if you don't have up to date backups of your router tables, it will take a long time to recover from an "oops".
Doing the wrong thing to router table(s) is the network equivalent of "sudo rm -rf /".
Re: GoDaddy outage caused by corrupted router tables
#19What is their 99.9% SLA liability going to be?
Re: GoDaddy outage caused by corrupted router tables
#20I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up.
Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make.
EDIT: And as someone else pointed out... their IP addresses could be pinged which further goes to disprove a routing issue. More than likely high traffic crashed one or more routers (THIS I have seen happen) and the live/saved configs didn't match. I'd put more money on something like this happening if it was router related.