Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoDaddy outage caused by corrupted router tables

godaddy.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: GoDaddy outage caused by corrupted router tables

#81

Post by peterwwillis »

druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Do you know how BGP works? There are easily 50 different ways routing problems can cause outages like this. More than likely there was a compound failure which can cause all kinds of retarded behavior, including different networks getting different kinds of traffic, to say nothing of a plain old network service on a single net being down.

Routers can "crash" for different reasons, but atypically due to high traffic. If you really wanted to fuck with someone you make one BGP change. Only newbs use DDoS's. (Which, Anonymous being newbs, would be their MO, but unlikely they could DDoS a connectionless resource record database)

Re: GoDaddy outage caused by corrupted router tables

#82

Post by darkr »

sophacles wrote: post #14
TomGullen wrote: post #4

I don't know much about hardware at all, but aren't routers fairly simple, time tested pieces of hardware? Can they really corrupt en-masse in this way?

Routers at this level aren't just scale-ups of your home wifi/nat box. They aren't even scale ups of the simple IP routers for a basic IT data-closet that manages subnets and whatnot (already much more complex by dealing with vlans and subnets and dmz and vpn issues). At the level of big networking company they are a truly complex beast. Just at the IP level they have to deal with (at the edges and across substantial…

Network geek here:

> Many are the tales of some small company misconfiguring their edge routers slightly (say a 1 char typo) and having the entire internet route through their T1, across their lan, out their backup T1.

This is what route filters are for. If you peer with someone and they advertise 0.0.0.0/0 or something equally ridiculous, and you accept this as a valid route then you deserve to fail (and then given a firm stare if you then proceed to advertise it to other peers).

A similar fail on the part of Telstra (http://bgpmon.net/blog/?p=554) was to blame for much of Australia dropping off the map earlier this year.

Also: > Just at the IP level they have to deal with (at the edges and across substantial WANS) BGP - a notoriously ugly and fragile protocol.

I take offence at this. 80% of BGP related issues are due to misconfiguration by a given party, 19% is due to bad or missing route filters and the other 1% is due to bugs in router software. The actual implementation of BGP v4, originally designed back in the early 90's isn't completely without it's issues and behavioural quirks (I'm looking at you, route flaps) but the theory/algorithm behind is a work of art, and has coped amazingly with explosive growth, and growth that's only going to increase with IPv6. Without it, there would be no HN

Re: GoDaddy outage caused by corrupted router tables

#83

Post by darkr »

gvb wrote: post #18
sophacles wrote: post #14

Earlier quoted context omitted.

Routers at this level aren't just scale-ups of your home wifi/nat box. They aren't even scale ups of the simple IP routers for a basic IT data-closet that manages subnets and whatnot (already much more complex by dealing with vlans and subnets and dmz and vpn issues). At the level of big networking company they are a truly complex beast. Just at the IP level they have to deal with (at the edges and across substantial…

...and if you don't have up to date backups of your router tables, it will take a long time to recover from an "oops". Doing the wrong thing to router table(s) is the network equivalent of "sudo rm -rf /".

router tables are a stationary woodworking machines in which a vertically oriented spindle of a woodworking router protrudes from the machine table and can be spun at speeds typically between 3000 and 24,000 rpm

excluding static routes (which are then usually advertised to other peers), routING tables are dynamically built and only exist in non-persistent memory.

having up to date backups of router configuration is another matter entirely

Re: GoDaddy outage caused by corrupted router tables

#84

Post by kijeda »

andreasvc wrote: post #43
kijeda wrote: post #16

The DNS is designed to provide resiliency to these kinds of problems by providing the ability to list multiple NS records located in different networks. It is standard practice for top-level domain operators and other high-activity domains to place their name servers in different networks to guard against these kinds of issues. When companies put all their name servers in the same network, they are removing the diver…

There are many other single points of failure besides network failure, such as pushing the wrong configuration. In fact it seems to me that it would be rather rare for a multi-homed datacenter to have a network failure.

If you have a routing issue, whether it is due to "corruption" or misconfiguration, having some of your name servers on an entirely different network (i.e. a different AS) with a different routing policy is not going to be affected.

Re: GoDaddy outage caused by corrupted router tables

#85

Post by count »

peterwwillis wrote: post #81
druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Do you know how BGP works? There are easily 50 different ways routing problems can cause outages like this. More than likely there was a compound failure which can cause all kinds of retarded behavior, including different networks getting different kinds of traffic, to say nothing of a plain old network service on a single net being down. Routers can "crash" for different reasons, but atypically due to high traffic.…

I'm familiar with BGP. I'm unfamiliar with how BGP has anything to do with me being able to ping their IP, but not get a response on UDP/53 or TCP/53 with any data in it.

Re: GoDaddy outage caused by corrupted router tables

#86

Post by druiid »

peterwwillis wrote: post #81
druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Do you know how BGP works? There are easily 50 different ways routing problems can cause outages like this. More than likely there was a compound failure which can cause all kinds of retarded behavior, including different networks getting different kinds of traffic, to say nothing of a plain old network service on a single net being down. Routers can "crash" for different reasons, but atypically due to high traffic.…

shrug I know how BGP works, yes. I think though from the symptoms of the issue it is going to end up being (If Godaddy is telling the truth about not being hacked), as you say, a compound failure. The exact cause of such a failure is left up to a truthful, full account of the outage being released. Further into this thread someone reported that an engineer from their side is going to release more information, so we'll see soon whom gets the prize :P.

Re: GoDaddy outage caused by corrupted router tables

#87

Post by barryrandall »

druiid wrote: post #42
dclowd9901 wrote: post #31

Earlier quoted context omitted.

Ex-GoDaddy employee here: Everyone there is on pins and needles at this point. Since Silverlake's investment in the company, many hatchets have dropped on jobs, and it's really the only decent tech firm in Phoenix to work at. My guess is that there is some hiney covering going on with this explanation, and the interim CEO has little cause to care too much about responsibility, since he'll likely be out before year's…

Ughh. That sounds about what I expected unfortunately. Sorry to hear this as any reasonable corporate structure understands 'shit' happens in the tech universe. Fix it as expediently as possible and move on, putting in place as much protection as possible to prevent it in the future. Do you think we'll see a true outage report or is this unlikely?

Based on the OP's statements about the company culture, is there an incentive for employees to be honest about their role in the outage? Do you think that enough Very Good Engineers stuck around post-acquisition to perform an independent root-cause analysis?

I've never known Go Daddy to make any public statements without testing them first. Whatever story gets released will likely be the one with the highest conversion rate.

Re: GoDaddy outage caused by corrupted router tables

#88

Post by jpdoctor »

dclowd9901 wrote: post #80
jpdoctor wrote: post #55

Earlier quoted context omitted.

> "It won't fail because of me." Leave now. It will be a black mark on your resume if you stay. Seriously.

> It will be a black mark on your resume if you stay. It's funny you say this. This is exactly the reason why I left, and exactly what I told HR when I left. Unsurprisingly, they had told me that others who had recently left gave similar sentiment.

Good move. Having a nose for when an organization will work for you among good organizations. (It will work against you in bad organizations "What?! You didn't stay to help turn it around?" SLP is most likely going to strip out whatever they can and then find somebody to unload it on.)

Re: GoDaddy outage caused by corrupted router tables

#89

Post by jpdoctor »

tnorthcutt wrote: post #61
jpdoctor wrote: post #55

Earlier quoted context omitted.

> "It won't fail because of me." Leave now. It will be a black mark on your resume if you stay. Seriously.

Ex-GoDaddy employee here

I should have been more clear that I was writing to those that were not ex.

Re: GoDaddy outage caused by corrupted router tables

#90

Post by sendos »

andreasvc wrote: post #70
46Bit wrote: post #57

Earlier quoted context omitted.

Got a link on that?

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-m... Not sure if those tweets were actually from before it happened.

I searched for those tweets mentioned in the Techcrunch article (first from GoDaddy and first from AnonymousOwn3r on this topic) and the time stamps are:

* GoDaddy: 10:35 AM - 10 Sep 12

* AnonymousOwn3r: 11:57 AM - 10 Sep 12

So, AnonymousOwn3r does not seem to have announced the attack before it happened.

Links to tweets:

* https://twitter.com/GoDaddy/status/245213898683318272

* https://twitter.com/AnonymousOwn3r/status/245234582205652992

Post reply on HN