Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Network Solutions' DNS was down

networksolutions.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Network Solutions' DNS was down

#51

Post by dsl »

esw wrote: post #21
pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.

FYI, Zerigo DNS is a joke.

They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707

Re: Network Solutions' DNS was down

#52

Post by ChuckMcM »

druiid wrote: post #49
pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was stil…

This has been my experience as well. Stupid bots being denied recursive requests. I modified my fail2ban script to autoban them which helped considerably.

Re: Network Solutions' DNS was down

#53

Post by druiid »

ChuckMcM wrote: post #52
druiid wrote: post #49

Earlier quoted context omitted.

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was stil…

This has been my experience as well. Stupid bots being denied recursive requests. I modified my fail2ban script to autoban them which helped considerably.

Well, fail2ban can help, but that is only going to be generally quick enough to keep up with the initial ramp-up of the attack I'd expect. Once the full attack gets going you're probably going to be seeing requests from thousands of hosts at any given time. I haven't specifically tested under that scenario with fail2ban, but I imagine that it would have trouble keeping up with such a flow.

Re: Network Solutions' DNS was down

#54

Post by rhizome »

druiid wrote: post #49
pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was stil…

This DDoS argument really is a premature optimization, and it's very simple to implement an external primary/secondary if the problem ever does occur.

Re: Network Solutions' DNS was down

#55

Post by ChuckMcM »

druiid wrote: post #53
ChuckMcM wrote: post #52

Earlier quoted context omitted.

This has been my experience as well. Stupid bots being denied recursive requests. I modified my fail2ban script to autoban them which helped considerably.

Well, fail2ban can help, but that is only going to be generally quick enough to keep up with the initial ramp-up of the attack I'd expect. Once the full attack gets going you're probably going to be seeing requests from thousands of hosts at any given time. I haven't specifically tested under that scenario with fail2ban, but I imagine that it would have trouble keeping up with such a flow.

My experience was that fail2ban mitigated it pretty much completely for my small server (which no doubt was only allocated a small number of bots to be attacked with). It took about an hour to build up a list of about 16,000 machines and then basically things were fine again.

Re: Network Solutions' DNS was down

#56

Post by dedward »

dholowiski wrote: post #19
pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

I thought that even if you self host - isn't it your registrar that points the dns records to your dns servers? Would someone with self-hosted DNS still be working in this situation? (doesn't the DNS query first go to network solutions, who refers it to your name servers?). I'm not trying to argue, I genuinely need to know the answer as I'll have to explain it to many other people, later today...

One aspect of Network Solutions business is managing some of the root zones. Those are the servers that ultimately tell the internet what other servers to go look at to resolve queries. These are the "root" nameservers. So while Network Solutions is in charge of the root zones themselves, and does run a bunch of root nameservers, there are other root nameservers run by Various organizations and collectively they can handle a massive amount of work. That is not what is being attacked. If somone DID successfully DDOS a large enough portion of the root nameserver infrastructure, the internet would largely grind to a halt. It is designed suitably sized, and run by large enough players that this is not likely.

The DNS service that is being attacked is the one they offer end users, usually along with domain registration. There is no particular reason you need to use their services when registering a domain with them, although, like most rigistrars, their website will make this deliberately confusing, and may require you to use their DNS service if you also want to use some of their other value-added services along with your registration (like email forwarding, websites, stuff like that).

Unless you are using other services from NetSol that require you to use their DNS, you are free to use whatever DNS infrastructure you like, including your own.

(If DDOS is a business concern, I'd go with someone big who deliberately does just DNS.. Neustar or one of those)

Re: Network Solutions' DNS was down

#57

Post by dedward »

druiid wrote: post #49
pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was stil…

To Rhizomes point: Yes, it's relatively easy to re-point them to somewhere else.. and if your business can tolerate the interruption that's fine.

Those changes, though, take time to cache out, and depending on your registrar's procedures and whatnot, you could be looking at hours or a couple of days to get things to stabilize back to normal; not to mention finding a new provider and getting things set up. You'll need ot make sure you have the zone data on hand ready to get set up with a new provider, and so on. You could have a procedure in place to do this, sure, with everything tested and validated, but at that point it's much easier to just have a DNS provider of the proper class who can handle these things for you. And keep in mind, if it's your domain they are trying to shut down, and not your DNS provider specifically, the attack will very likely just follow the change, leaving you in quite a mess.

So the short answer: self-host if it's not revenue impacting. If there is money at stake, spend a little bit for a top-notch provider.

Re: Network Solutions' DNS was down

#58

Post by dedward »

thejosh wrote: post #39
nevir wrote: post #36

Earlier quoted context omitted.

Well, what happens if someone doesn't like your registrar and DDoS' them? Do both.

You would hope that a registrar would have better bandwidth and/or capabilities to protect against that then what you would generally have for your server(s).

You would.. but the business of being a registrar is bout managing zone records to be served out by the root/regional nameserver infrastructure, which your registrar may not even have much of a part in.

If they run additional business in hosting DNS, websites, email, and so on, as is common - that's really got nothing to do with the setup they need to act as a registrar.

Re: Network Solutions' DNS was down

#59

Post by dedward »

pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

What's important is that you treat the DNS management aspect of your business as a proper thing to be managed by itself. Whether you host it yourself, or contract it out to a service that specifically does this for you is a matter of risk tolerance and budget.

A common setup is a private nameserver that you host yourself, wherever, that isn't publicly used. You then contract UltraDNS/Neustar or their brethren and set them up as your public servers, doing zone transfers from your private ones. You get the advantages of direct management of your zones, and the global scale infrastructure needed to largely eliminate nameserver issues.

Re: Network Solutions' DNS was down

#60

Post by dedward »

mikegirouard wrote: post #10
esmale wrote: post #8

I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.

Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP. It doesn't matter how long I do this... DNS problems always get me.

Have you thought about setting up with an external DNS monitoring service? It's cheap and really comes in handy over the years.
Post reply on HN