Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoDaddy outage caused by corrupted router tables

godaddy.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: GoDaddy outage caused by corrupted router tables

#41

Post by breck »

highfreq wrote: post #23

How can they claim 99.999% uptime, when they just had several hours of service outage? I'm not sure how long they've been providing DNS hosting, but by the most generous assumption this would be the entire 15 years of their existence. 99.999% allows them about 1.3 hours of outage in 15 years.

> "Throughout our history, we have provided 99.999% uptime in our DNS infrastructure"

I think he's implying "until yesterday".

I've used GoDaddy for 10 years or so, and this is the first DNS outage I can remember.

That's an impressive record, imo.

(Personally I've largely switched off GoDaddy to Namecheap now, but not because of uptime, rather because I grew tired of the sucky GoDaddy website and sucky APIs)

Re: GoDaddy outage caused by corrupted router tables

#42

Post by druiid »

dclowd9901 wrote: post #31
staunch wrote: post #10

Based on a long history of working in datacenters I'd bet someone misconfigured something and later claimed it was "corrupted" to save their ass - happens all the time. It's just so simple to make very confusing and damaging mistakes in a complicated network. I wouldn't be surprised to hear that GoDaddy's corporate culture wouldn't respond well to someone admitting to a mistake this damaging.

Ex-GoDaddy employee here: Everyone there is on pins and needles at this point. Since Silverlake's investment in the company, many hatchets have dropped on jobs, and it's really the only decent tech firm in Phoenix to work at. My guess is that there is some hiney covering going on with this explanation, and the interim CEO has little cause to care too much about responsibility, since he'll likely be out before year's…

Ughh. That sounds about what I expected unfortunately.

Sorry to hear this as any reasonable corporate structure understands 'shit' happens in the tech universe. Fix it as expediently as possible and move on, putting in place as much protection as possible to prevent it in the future.

Do you think we'll see a true outage report or is this unlikely?

Re: GoDaddy outage caused by corrupted router tables

#43

Post by andreasvc »

kijeda wrote: post #16

The DNS is designed to provide resiliency to these kinds of problems by providing the ability to list multiple NS records located in different networks. It is standard practice for top-level domain operators and other high-activity domains to place their name servers in different networks to guard against these kinds of issues. When companies put all their name servers in the same network, they are removing the diver…

There are many other single points of failure besides network failure, such as pushing the wrong configuration. In fact it seems to me that it would be rather rare for a multi-homed datacenter to have a network failure.

Re: GoDaddy outage caused by corrupted router tables

#44

Post by mongol »

SilasX wrote: post #40

Semi-OT: Why is it so hard to find yesterday's highly-rated GoDaddy outage discussion? Neither sorting by relevance nor recency nor points will find it. Or maybe there wasn't one?

I don't know why it is hard but I picked it up from my browser history: http://news.ycombinator.com/item?id=4500993

Re: GoDaddy outage caused by corrupted router tables

#45

Post by andreasvc »

overworkedasian wrote: post #11

I wasnt really paying attention to the outage, but if it was indeed a routing issue, then you shouldnt have been able to reach any godaddy ip address. ICMP/traceroutes would have failed and showed the error.

You're forgetting internal routing; e.g. unreachable database server.

Re: GoDaddy outage caused by corrupted router tables

#46

Post by andreasvc »

sendos wrote: post #7
hbz wrote: post #3

"The service outage was not caused by external influences. It was not a "hack" and it was not a denial of service attack." So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? The router tables just corrupted themselves?

> So Anonymous0wn3r or whoever was just claiming responsibility for something they had no hand in? Yeah, with a name/handle like Anonymous0wn3r they sound very trustworthy. If they claimed they had a hand in something, it must be true.

But didn't he announce the attack before-hand?

Re: GoDaddy outage caused by corrupted router tables

#47

Post by omd »

nettdata wrote: post #34
druiid wrote: post #29

Earlier quoted context omitted.

None of that is necessarily incorrect... but per their news release 'corrupted router data tables' (their words) were the issue. I can't read too much into that, but that still doesn't change the fact that DNS wasn't resolving for a while after they made their Verisign change (for clients), yet their website was resolved with this change. You are correct that I don't know the details of their internal network and I n…

I can imagine that they'd understandably work to get their own site/etc up and running first as the priority, as a manual "hack". After all, it's the main page everyone would be going to for information on what's going on. After that, coming up with an automated process for migrating what must be a shit-ton of zone information to another system must have taken some time. I have no idea what their specific solution wa…

I don't see it as defending GoDaddy at all, quite the opposite. I would be more reassured if it was an unexpected massive DDoS which they weren't prepared for but one which they might prepare for in the future.

The way it's described now is a weakness in their infrastructure of which I wonder if it's possible to prevent this from happening again.

Re: GoDaddy outage caused by corrupted router tables

#48

Post by jauer »

druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

Router bugs aren't unheard of. There was the Juniper MX bug that caused multiple outages for Level3 & Time Warner Cable. That was supposedly just a bad pattern of route injections and withdrawls.

That said I agree that Godaddy's handling and RFO doesn't smell right.

Re: GoDaddy outage caused by corrupted router tables

#49

Post by SilasX »

mongol wrote: post #44
SilasX wrote: post #40

Semi-OT: Why is it so hard to find yesterday's highly-rated GoDaddy outage discussion? Neither sorting by relevance nor recency nor points will find it. Or maybe there wasn't one?

I don't know why it is hard but I picked it up from my browser history: http://news.ycombinator.com/item?id=4500993

Your confirmation of my sanity is appreciated :-)
Post reply on HN