Live data from Hacker News

Ask HN: DNS redundancy, how to do it right?

news.ycombinator.com

1–10 of 32 posts

Ask HN: DNS redundancy, how to do it right?

#1
I hope I'm doing this right, since it's my first submission. It is a question directed to sysadmins of HN:

How do I reach nameserver redundancy?

Right now our provider is getting DdoS'ed, so my employer is not reachable by mail, web etc. If I do a whois on the affected domain, I'll get multiple nameservers (which the provider owns).

Looks like this:

nserver ns01.provider.tld nserver ns02.provider.tld nserver ns03.provider.tld nserver ns04.provider.tld nserver ns05.provider.tld

Actually two questions arise from this:

- Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

- Is it a good idea to simply increase the TTL of the important A/MX-Records? Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? (assumed I have a TTL of 3 hours for example)

Thankfully, I'm not the CTO, but since he mentioned to me that this happens regularly to the provider (being DdoSed), it got me really curious what the right mitigation to being unreachable is.

Re: Ask HN: DNS redundancy, how to do it right?

#2
Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

Do you mean authoritative secondary replicas? If so, that is not uncommon. If your DNS provider is being targeted and your company is not then your DNS servers will still respond and a percentage of clients will try them. While root servers allow 10 records, anything beyond 4 will become less useful as different resolvers cap the number of NS records they will try. You can look up which OS/resolver has what behavior and then make a decision based on what OS most of your customers use. Amazon for example uses batches of 5 anycast records. If your commercial DNS provider is also Anycast, theny one could use 2 of those records and 2 of your own company hosted DNS just fine.

Look into how your commercial DNS provider handles zone transfers then set up a couple decent servers that uptake all the zones you want redundancy for. Just know there is no concept of priority meaning what order they are listed in the root servers does not matter. Whatever servers you set up will need to take a percentage of the traffic your commercial provider is absorbing. If doing this on a VPS provider I would suggest Vultr as they support Anycast meaning you can spin up many VM's to handle the load and still only have a couple public IP addresses without any load-balancer bottlenecks.

Is it a good idea to simply increase the TTL of the important A/MX-Records? Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? (assumed I have a TTL of 3 hours for example)

There are pros and cons to high TTL's depending on how your organization handles changes, failovers, etc... There are some discussions on the web about these pros and cons, too many to name here. It is also important to understand how clients actually cache high TTL's. For example, some clients will cap NS TTL to 86400 seconds regardless of how high they are and some clients will cap A TTL to 1 or 3 days. Then there is the factor of recursive server memory and end-users. ISP caches will expire records much faster regardless of TTL due to memory pressure. Each ISP and public DNS server handles this a little differently. So a high TTL can sometimes help assuming your infrastructure does not depend on being able to fail over things fast and that you are not planning on changing MX end-points. This requires some foresight into how one architects their infrastructure to fully recognize the benefits from higher TTL without incurring operational risk.

I am testing 1.1.1.1 right now and it took many requests to finally get my records cached on all their nodes, so if your domain is popular enough they may be useful.

I suppose that was a long-winded way of saying, "It depends". You should meet with your infrastructure team and think through what systems depend on having a low TTL and keep those low. For anything else a higher TTL is probably fine.

[Edit] It sounds like maybe you were just asking about recursive servers so most of this doesn't even apply.

Some places to browse for more detailed answers would be StackExchange [1] ServerFault [2] SuperUser [3] Just be sure to lurk a long time before asking questions. They are particular about how questions are formatted, how on-topic they are for the particular forum and if one has done an exhaustive search for existing answers.

[1] - https://unix.stackexchange.com/

[2] - https://serverfault.com/

[3] - https://superuser.com/

Re: Ask HN: DNS redundancy, how to do it right?

#3
> Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour?

1.1.1.1 is virtually impossible to DDoS, because it’s anycasted in _a lot_ of places, and Cloudflare has the capacity to mitigate the largest of DDoS attacks.

> - Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary?

This would be a caching NS. It's not a bad idea, especially if it can automatically forward to a different service provided one is down, but you might as well just use 1.1.1.1. I've never seen 1.1.1.1 down.

Re: Ask HN: DNS redundancy, how to do it right?

#4

> Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? 1.1.1.1 is virtually impossible to DDoS, because it’s anycasted in _a lot_ of places, and Cloudflare has the capacity to mitigate the largest of DDoS attacks. > - Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that…

Actually this is a much better idea than hosting my own namserver.

Re: Ask HN: DNS redundancy, how to do it right?

#5

Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary? Do you mean authoritative secondary replicas? If so, that is not uncommon. If your DNS provider is being targeted and your…

Thanks a lot for the detailed answer. I think having a higher TTL + using 1.1.1.1 as nameserver would be a good idea. I'll let the 'infrastructure team' know about this suggestion

Re: Ask HN: DNS redundancy, how to do it right?

#6

> Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? 1.1.1.1 is virtually impossible to DDoS, because it’s anycasted in _a lot_ of places, and Cloudflare has the capacity to mitigate the largest of DDoS attacks. > - Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that…

I've seen some do this "stealth secondary" architecture before and it's helpful for local zones. Sometimes they make the local recursive resolver be a secondary of the local zone, but don't include the recursive resolver in the NS set of the zone. That way the recursive resolver has a full copy of local zones for responses and doesn't have to ask an authoritative server (technically it is one, but only known to hosts that use it as their local recursive resolver). This only works if you don't use public resolvers.

Re: Ask HN: DNS redundancy, how to do it right?

#8
post #4

> Will for example, 1.1.1.1 still resolve my domain correctly, even if my providers nameserver is down for an hour? 1.1.1.1 is virtually impossible to DDoS, because it’s anycasted in _a lot_ of places, and Cloudflare has the capacity to mitigate the largest of DDoS attacks. > - Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that…

Actually this is a much better idea than hosting my own namserver.

Yes. The most important thing here is that 1.1.1.1 is announced in seriously a lot of places. I just went to Hurricane Electric's looking glass, several times picked 3 routers at random and put in 1.1.1.1 as the destination. All of them returned it as the 2nd hop, meaning it's most likely announced in the same exact datacenter as where HEs router is located.

That gives you all the redundancy you'll ever need. Of course, you need to rely on Cloudflare to not mess it up. But you're not going to do better than CF.

Re: Ask HN: DNS redundancy, how to do it right?

#9
post #5

Is it a good idea to setup my own nameserver which basically just "copies" the entries from my current provider and specify it (wherever that may be). By doing this I won't have to maintain 2 different NS, only the one from the provider since the 'secondary' will simply be a copy of the primary? Do you mean authoritative secondary replicas? If so, that is not uncommon. If your DNS provider is being targeted and your…

Thanks a lot for the detailed answer. I think having a higher TTL + using 1.1.1.1 as nameserver would be a good idea. I'll let the 'infrastructure team' know about this suggestion

If using the public DNS servers to resolve things then certainly have a few of them. 1.1.1.1 and 8.8.4.4 in case CF has problems.

[Edit] Testing 1.1.1.1 it seems I have finally built cache on all their nodes. It took about 25 to 30 requests. Second test took only a few requests for a different record, same domain. Now I am more curious about their back-end.

If you have your own recursive servers in your datacenter then you can entirely control how many things are cached and how that cache behaves. Unbound is a really good option for this as it is fast, has controls around memory, threads, min/max TTL and you could even push your authoritative zones to the edge Unbound nodes if desired so that those records never expire. Some people take this a step further in their datacenter and have Unbound running on every instance to keep response latency low and handle upstream recursive fail-over better than the OS resolver does.

Another benefit to running your own caching servers is that you can purge records that you know are out of date during outages.

Re: Ask HN: DNS redundancy, how to do it right?

#10
"Right now our provider is getting DdoS'ed, so my employer is not reachable by mail, web etc."

"our" here might suggest that your provider is also your employer's provider, and that your employer is not reachable in general by anyone by mail, web, et cetera. But reading the rest of your message makes me think that perhaps you're saying that your personal provider is being attacked, not your employer's, and therefore you can't reach your employer's mail, web, et cetera. Is that the case?

Is the attack just taking out your provider's DNS servers? If so, then just run your own recursive resolver. It's literally as easy as setting up BIND on any machine on your network with a default configuration file that does the barest minimum. Clients on the same subnet will be able to query it without problems.

There's no reason, nor advantage, to running anything using your provider's DNS servers when you have your own, particularly when their DNS servers can be taken down so easily.

Post reply on HN