GoDaddy outage caused by corrupted router tables • Hacker News Board

Re: GoDaddy outage caused by corrupted router tables

#91

Post by Osiris » Tue, Sep 11, 2012, 11:43 PM UTC

dclowd9901 wrote: post #78

citricsquid wrote: post #39

Earlier quoted context omitted.

can you clarify what you mean by: > it's really the only decent tech firm in Phoenix to work at. and then: > Horrible, horrible, backward-ass culture. what part of the company is good if not the culture?

There are many tech (and non-tech) companies in the Phoenix area. Your choice, as always, is to work at a place where the importance of the technology and the talent of your engineers and designers is recognized, or work in PetSmart's IT/Web department (where you're constantly fighting budget and recognition battles). So should you opt to go into tech, GoDaddy is the largest and best-paying, and has fairly decent ben…

I've only been here less than a year so I haven't been exposed to that. What I've observed is that the culture is very compartmentalized in the sense that there are lots of small teams that each have a specific product they work on. Each team basically has it's own culture and way of doing things.

That isn't good for standards/best practices, but it does mean that teams have a lot of leeway to do things pretty much however they want. My team has a ton of flexibility on how we code, test, and deploy, though we have to work within some red tape imposed by other teams that have their own way of doing their things.

In my day to day work I'm actually pretty free to work how I want without being pushed by people up the food chain. That could be because the product I work on is a supplementary product and not a 'core' product, though.

Re: GoDaddy outage caused by corrupted router tables

#92

Post by napillo » Wed, Sep 12, 2012, 12:18 AM UTC

oasisbob wrote: post #79

On the outages mailing list[1], Mike Dob (GoDaddy Network Engineering Manager) just added more details, saying: > It was BGP related and more details should be posted today [1] http://puck.nether.net/mailman/listinfo/outages

BGP? So it was a bad route injected? They should have had someone there looking for that, that certainly wouldn't take them down for 5 hours.

Re: GoDaddy outage caused by corrupted router tables

#93

Post by dkopack » Wed, Sep 12, 2012, 12:19 AM UTC

kevincennis wrote: post #50

For anyone interested, the person who claimed responsibility for this is tweeting about GoDaddy's response: https://twitter.com/AnonymousOwn3r/status/245568841160196096

Just checked his twitter feed, looks like he's attempting to claim source from an old open source project as part of the claimed hack...credibility gone.

Re: GoDaddy outage caused by corrupted router tables

#94

Post by dingdingpop » Wed, Sep 12, 2012, 6:58 AM UTC

Their engineer claims it was an issue with BGP (http://permalink.gmane.org/gmane.org.operators.isotf.outages...).

BGPlay (http://bgplay.routeviews.org/) does not show anything indicative in the BGP default-free table (what the Internet sees), as abnormal or misconfigured. While there could be iBGP issues, like others have stated there was (intermittent) connectivity by IP during the outage.

It's both bullshit PR and more importantly spreading disinformation to save face. Why?

A security breach would instill customer fear and generate negative press. Customers would leave by the droves.

A DoS/DDoS displays that GoDaddy has inadequate infrastructure while competitors such as CloudFare actually do. Furthermore, why would a company that pisses off the Internet be appealing to anyone? Again it will generate negative/bad press, and customers will leave by the drove.

Spreading disinformation by claiming it was either a human error or equipment fault? From a company perspective this is actually the best option. Just provide generous service credit to your customers, you may generate positive press, you will gain customer goodwill and regain their confidence. This is GoDaddy's best option.

Until they provide actual details with proof that it was a misconfiguration or hardware fault, I will continue to call bullshit. Too many factors don't add up, especially the publicly available data which monitors the BGP DFT on the Internet.

The two conjectures that seem plausible so far is the SQL injection in their web interface for DNS and/or a DoS/DDoS attack.

Re: GoDaddy outage caused by corrupted router tables

#95

Post by peterwwillis » Wed, Sep 12, 2012, 3:35 PM UTC

count wrote: post #85

peterwwillis wrote: post #81

Earlier quoted context omitted.

Do you know how BGP works? There are easily 50 different ways routing problems can cause outages like this. More than likely there was a compound failure which can cause all kinds of retarded behavior, including different networks getting different kinds of traffic, to say nothing of a plain old network service on a single net being down. Routers can "crash" for different reasons, but atypically due to high traffic.…

I'm familiar with BGP. I'm unfamiliar with how BGP has anything to do with me being able to ping their IP, but not get a response on UDP/53 or TCP/53 with any data in it.

Off the top of my head? One network they multihome had a weird packet loss only experienced by DNS and other services, so they tried to cut the routes over to the second network, but packets were still getting sent to the first network (which had DNS disabled but ICMP enabled on the hosts) and further router fuckage prevented them from switching back easily. Hell, they probably just couldn't get their BGP to propagate once they made the first change.

If you go with 'router tables' being the culprit, they probably had a core router that maxed out its RAM when they added another router in place, but they had already moved a part of the network that housed DNS by the time the routers synced and RAM filled from too many BGP lists to sort. You can still ping 'hosts' (which are I am almost certain a hardware load balancer and not an actual DNS host) while the DNS traffic is going nowhere because the backend DNS services were moved. Would take a couple hours to unfuck all of that.

Re: GoDaddy outage caused by corrupted router tables

#96

Post by sophacles » Wed, Sep 12, 2012, 3:59 PM UTC

darkr wrote: post #82

sophacles wrote: post #14

Earlier quoted context omitted.

Routers at this level aren't just scale-ups of your home wifi/nat box. They aren't even scale ups of the simple IP routers for a basic IT data-closet that manages subnets and whatnot (already much more complex by dealing with vlans and subnets and dmz and vpn issues). At the level of big networking company they are a truly complex beast. Just at the IP level they have to deal with (at the edges and across substantial…

Network geek here: > Many are the tales of some small company misconfiguring their edge routers slightly (say a 1 char typo) and having the entire internet route through their T1, across their lan, out their backup T1. This is what route filters are for. If you peer with someone and they advertise 0.0.0.0/0 or something equally ridiculous, and you accept this as a valid route then you deserve to fail (and then given…

Thanks for your views on BGP - a lot of my knowledge of it comes from post-incident beers with our network grey-beards when I worked at an ISP, so my views are probably somewhat biased. There is nothing like a network explosion at an ISP to get people ranting about BGP, but I'll admit that the ranting is from a place of anger and frustration and largely venting rather then a fair technical assessment.

Re: GoDaddy outage caused by corrupted router tables

#97

Post by dclowd9901 » Wed, Sep 12, 2012, 9:10 PM UTC

Osiris wrote: post #91

dclowd9901 wrote: post #78

Earlier quoted context omitted.

There are many tech (and non-tech) companies in the Phoenix area. Your choice, as always, is to work at a place where the importance of the technology and the talent of your engineers and designers is recognized, or work in PetSmart's IT/Web department (where you're constantly fighting budget and recognition battles). So should you opt to go into tech, GoDaddy is the largest and best-paying, and has fairly decent ben…

I've only been here less than a year so I haven't been exposed to that. What I've observed is that the culture is very compartmentalized in the sense that there are lots of small teams that each have a specific product they work on. Each team basically has it's own culture and way of doing things. That isn't good for standards/best practices, but it does mean that teams have a lot of leeway to do things pretty much h…

Classic engineer. I'm talking about across a product. It's true that within your core group, you are given leeway, but the buck stops there.