Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Network Solutions' DNS was down

networksolutions.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Network Solutions' DNS was down

#41

Post by mmmooo »

colmmacc wrote: post #40
area51org wrote: post #24

Earlier quoted context omitted.

Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.

(I work on Route 53). We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hai…

Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.

Re: Network Solutions' DNS was down

#42

Post by smountcastle »

area51org wrote: post #24
simonsarris wrote: post #20

Honest question from a networking beginner: So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues. If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?

Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.

It depends on the implementation of the resolver, but in general when no information is known about a nameserver, the resolver will pick one at random. Overtime, it'll keep track of the latency and pick one (usually randomly within a band specifying a range of latencies). See this paper for more info: https://www.dns-oarc.net/files/workshop-201203/OARC-workshop...

Re: Network Solutions' DNS was down

#43

Post by colmmacc »

mmmooo wrote: post #41
colmmacc wrote: post #40

Earlier quoted context omitted.

(I work on Route 53). We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hai…

Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.

I haven't read the bind source in a long time, but my best guess is that it's in memory.

http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h...

Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.

Re: Network Solutions' DNS was down

#45

Post by mmmooo »

colmmacc wrote: post #43
mmmooo wrote: post #41

Earlier quoted context omitted.

Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.

I haven't read the bind source in a long time, but my best guess is that it's in memory. http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h... Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.

interesting, not a ton of details of how. off to the source I go, thanks. sounds like it assigns a 'score' of some sort to each authoritative ns.

Re: Network Solutions' DNS was down

#46

Post by throwit1979 »

simonsarris wrote: post #20

Honest question from a networking beginner: So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues. If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?

In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.

Re: Network Solutions' DNS was down

#47

Post by muraiki »

Tenhundfeld wrote: post #44

TechZone360 (ugh, so many ads) article with a little more information: Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/...

A comment in the article pointed out that this one seems very similar to a Cisco article from a month ago: http://blogs.cisco.com/security/hijacking-of-dns-records-fro...

Re: Network Solutions' DNS was down

#49

Post by druiid »

pilif wrote: post #16

This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers. Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an un…

You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was still enough to saturate the connection with our provider at the time.

This is only one of many possible scenarios with self-hosted DNS. Essentially there are services out there (generally Anycast are best) that do only DNS and have had very high up-time. I'd suggest if you are anything over a 'small' target to look into these services. Sadly it's one of the few things I basically suggest not self-hosting at this point simply due to the unrealistic requirements to scale it yourself under attack scenarios.

Re: Network Solutions' DNS was down

#50

Post by dsl »

mmmooo wrote: post #45
colmmacc wrote: post #43

Earlier quoted context omitted.

I haven't read the bind source in a long time, but my best guess is that it's in memory. http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h... Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.

interesting, not a ton of details of how. off to the source I go, thanks. sounds like it assigns a 'score' of some sort to each authoritative ns.

BIND assigns a score based on every IP it talks to and a hit rate. (so if ns1.foo.com resolves to 3 IPs, each is tracked individually, same if ns1.bar.com and ns1.baz.com are the same IP) It is a pretty small cache, maybe like 1024?

This is all from fuzzy 5 year old memory.

Post reply on HN