Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoDaddy outage caused by corrupted router tables

godaddy.com

Post reply on HN
Print view View original

61–70 of 97 posts

Re: GoDaddy outage caused by corrupted router tables

#61

Post by tnorthcutt »

jpdoctor wrote: post #55
dclowd9901 wrote: post #31

Earlier quoted context omitted.

Ex-GoDaddy employee here: Everyone there is on pins and needles at this point. Since Silverlake's investment in the company, many hatchets have dropped on jobs, and it's really the only decent tech firm in Phoenix to work at. My guess is that there is some hiney covering going on with this explanation, and the interim CEO has little cause to care too much about responsibility, since he'll likely be out before year's…

> "It won't fail because of me." Leave now. It will be a black mark on your resume if you stay. Seriously.

Ex-GoDaddy employee here

Re: GoDaddy outage caused by corrupted router tables

#62

Post by larrys »

druiid wrote: post #37
nettdata wrote: post #34

Earlier quoted context omitted.

I can imagine that they'd understandably work to get their own site/etc up and running first as the priority, as a manual "hack". After all, it's the main page everyone would be going to for information on what's going on. After that, coming up with an automated process for migrating what must be a shit-ton of zone information to another system must have taken some time. I have no idea what their specific solution wa…

Heh, yeah. It is a bit difficult to interpret PR speak (and I have had to correct our guy before). I think perhaps the takeaway from here is to not trust what is being said, go with your gut... and move any services off GoDaddy ;). Would be nice if like Google or Amazon they would release a real post-mortem post. Even if it's an internal 'uh-oh' I trust companies that are willing to admit to mistakes.

"Would be nice if like Google or Amazon they would release a real post-mortem post."

Possible but highly unlikely. Godaddy is "old school" which means they will release as little info as necessary and move on. They aren't interested in the hacker community. Their primary market is SMB's.

Re: GoDaddy outage caused by corrupted router tables

#64

Post by larrys »

omd wrote: post #47
nettdata wrote: post #34

Earlier quoted context omitted.

I can imagine that they'd understandably work to get their own site/etc up and running first as the priority, as a manual "hack". After all, it's the main page everyone would be going to for information on what's going on. After that, coming up with an automated process for migrating what must be a shit-ton of zone information to another system must have taken some time. I have no idea what their specific solution wa…

I don't see it as defending GoDaddy at all, quite the opposite. I would be more reassured if it was an unexpected massive DDoS which they weren't prepared for but one which they might prepare for in the future. The way it's described now is a weakness in their infrastructure of which I wonder if it's possible to prevent this from happening again.

"The way it's described now is a weakness in their infrastructure"

Godaddy has plenty to lose by f-ing up. And to my knowledge (as a somewhat small competitor; I'm just pointing that out so my thoughts are taken in context) they have a fairly robust system (anecdotal) for the amount of data they manage. My issues with godaddy (as a competitor) were always on the sell side, the issues of constantly selling you things you don't need etc. Technically I really didn't have any issues with them.

Re: GoDaddy outage caused by corrupted router tables

#66

Post by SeanDav »

druiid wrote: post #20

I find this extremely suspicious (I.E. knowing routers, I call bullshit). The change to the Verisign anycast DNS service which I noted yesterday in another thread... brought godaddy.com back up, yet did not result in bringing other DNS services back up. Someone is lying here in my opinion. I hope I'm proven wrong because this is a terrible excuse for the company to make. EDIT: And as someone else pointed out... their…

It would certainly be in GoDaddy's interests to claim a technical fault, rather than admit to a hack, which implies lax security.

Re: GoDaddy outage caused by corrupted router tables

#67

Post by rhizome »

citricsquid wrote: post #39
dclowd9901 wrote: post #31

Earlier quoted context omitted.

Ex-GoDaddy employee here: Everyone there is on pins and needles at this point. Since Silverlake's investment in the company, many hatchets have dropped on jobs, and it's really the only decent tech firm in Phoenix to work at. My guess is that there is some hiney covering going on with this explanation, and the interim CEO has little cause to care too much about responsibility, since he'll likely be out before year's…

can you clarify what you mean by: > it's really the only decent tech firm in Phoenix to work at. and then: > Horrible, horrible, backward-ass culture. what part of the company is good if not the culture?

possibly the subcultures that emerge to complain about the backward-ass mainstream within the company, i.e. having something to gripe about with your coworkers is a bonding force.

Re: GoDaddy outage caused by corrupted router tables

#68

Post by zbuc »

kevincennis wrote: post #50

For anyone interested, the person who claimed responsibility for this is tweeting about GoDaddy's response: https://twitter.com/AnonymousOwn3r/status/245568841160196096

This guy seems to be full of crap to me.

Did you see the Pastebin he posted?

"cool wow godaddy is so safe lol i just found 51 vulnerabilities http://pastebin.com/EdCDaqQy "

It's a bunch of form POST values he got 302 redirects back from? No indication of XSS at all. What's this dude going on about?

Re: GoDaddy outage caused by corrupted router tables

#69

Post by napillo »

staunch wrote: post #10

Based on a long history of working in datacenters I'd bet someone misconfigured something and later claimed it was "corrupted" to save their ass - happens all the time. It's just so simple to make very confusing and damaging mistakes in a complicated network. I wouldn't be surprised to hear that GoDaddy's corporate culture wouldn't respond well to someone admitting to a mistake this damaging.

Well, I don't know where you worked, but routers being misconfigured never took down any of my networks for longer than a couple hours. I've always had backups of my configurations that I could restore at a moment's notice.

Re: GoDaddy outage caused by corrupted router tables

#70

Post by andreasvc »

46Bit wrote: post #57
andreasvc wrote: post #46

Earlier quoted context omitted.

But didn't he announce the attack before-hand?

Got a link on that?

http://techcrunch.com/2012/09/10/godaddy-outage-takes-down-m...

Not sure if those tweets were actually from before it happened.

Post reply on HN