[deleted]
Ask HN: DNS redundancy, how to do it right?
21–30 of 32 posts
Re: Ask HN: DNS redundancy, how to do it right?
#22You want multiple NS records containing hostnames of nameservers from two or more providers on independent infrastructure (i.e. not two providers that are both hosted on, for example, AWS' compute).
I would advise against running your own nameserver unless you have confidence in your ability to operate it correctly.
You can increase the TTLs if you don't anticipate record data changing frequently, or are able to tolerate delays in your DNS record changes being served (until the cached answers expire).
Choice of resolver (e.g. 1.1.1.1, 8.8.4.4) is out of your control (except of course, on your own devices and machines). Increasing the TTLs may improve robustness, assuming that your clients' resolvers are well-behaved and respect TTLs [0].
Re: Ask HN: DNS redundancy, how to do it right?
#23If all of your records are traditional static records, DNS redundancy is pretty simple. DNS includes AXFR for secondary servers to pull the zone data from the primary. You can set up your own primary and use commercial services as secondary, or use one service as primary and another as secondary, etc. If the primary is ddosed, you might not be able to make updates if the secondaries can't connect for axfr, but often that happens on different ips than the public service ips, so there's a chance it still works.
But, it gets a lot trickier if you use any sort of DNS based load balancing or other trickery. Then you've got to set up both services as close to the same as you can and cross your fingers; there's tools out there for that, I think terraform can do it, but there's more focused tools as well. This is a good practice, but it's hard, so it's usually not done.
Top tier DNS services rarely get (succesfully) DDoSed or have other service outages, but it does happen.
Re: Ask HN: DNS redundancy, how to do it right?
#24Earlier quoted context omitted.
However there is value in outsourcing to two companies, and putting nameservers from both in your NS record. That way if one of them suffers a service outage for whatever reason, your users won't even notice.
Yep, we also have two. Not a DNS expert but wouldn't half the clients (or their resolvers) stop working if one of your two providers are down? But you can recover by removing the provider that is down from your NS, assuming that the TTL is not too long.
Generally, recursive resolvers will retry queries multiple times and use all the NS records as needed.
You shouldn't have a broken NS record if you can avoid it, and it may add a little bit of delay, but things will still work.
Some recursive resolvers also keep statistics about authoritative servers, and when a domain had multiple NS records, the recursive resolver will prefer the option that has had the best response time in the recent past. That helps hide broken servers.
Re: Ask HN: DNS redundancy, how to do it right?
#25In this case, the vendor probably needs to be replaced. That's the easy fix. The higher TTL will help as it won't be querying them, but then making changes will be a pain in the future, until unless you pre-plan.
These are some things I am have though about for managing DNS. Any other recommendations would be great.
1) Separate Registrar from the source of the DNS records
2) Multiple NS records on the registrar
3) Multiple vendors for DNS records with Terraform sync between the two vendors
4) Limit A records or point A Records to CNAMES
5) Low TTL times for NS records, risky if you are hacked though or make a bad change
Re: Ask HN: DNS redundancy, how to do it right?
#26Simple answer:
1) Spend money.
2) Get a different provider.
Beyond that, get free help or advice. I guess that's what you're attempting now.
Let's start with spending money. You need to convince somebody that it's worth it. (Know your assets, know your risks, etc.) If it's not worth it or it's killing the company, and they won't do anything about it except to send you out to beg for help: get another job. Seriously.
As for the provider and "DOS": put their balls in a vice. What kind of DOS, exactly? What mitigations do they employ, exactly? Are they visibly seeking advice and assistance? Where's their outage page? I (charitably) assume you're being coy because you don't know... not because you're just being coy. Get the facts; cache la poudre; name and shame them.
Who else is affected? Band together, share notes and intelligence. Openly. Fully. Go read some of the DNS server mailing lists and dns-ops. If you can't swim in those waters, go home. Hide, and hope they go away.
Most of the answers here are akin to poking a dead beached whale: "smells bad!" "look, there's its liver!" "that's the blowhole, that's how it breathes" "looks like a propellor strike": factual, but not gonna help the whale.
I'm baffled by the premise of your question: exactly how does this lead to needing to do "redundancy" correctly? Is the provider not doing it correctly? No evidence is provided to support the assertion.
Reachability and services: there are a lot of tactical measures depending on how services are hosted. Mirroring the domain is a tool in the arsenal, depending on your line of business and communications needs (monitoring a SCADA system for emails to make sure the nuke doesn't melt down is different from some rando wanting to return a party dress).
"multiple nameservers": anycast is a thing.
Ummm... mirroring the domain ("just 'copies'": WTF?) IS maintaining an NS.
I'm not going to assume anything about your TTL. Name the domain. Tell me the TTL. Let me confirm it. ("simply increase the TTL": WTF? I think "simple" is the important word there. None of this is simple.)
--
m3047 | FWM6, internet plumber
Re: Ask HN: DNS redundancy, how to do it right?
#27My $dayjob is at a domain registry operator.
Find out if your provider will allow you to add your own nameservers and allow zone transfers to them. Most will but find out because you REALLY don't want to synchronize changes manually.
You don't need anything fancy, add one or two of your own nameservers. Something at Hetner, OVH, DO, AWS etc are all fine. You only need small basic Linux box with Knot, NSD, or Bind installed and a gig or 2 of memory.
Don't worry about Powerdns if you only have a couple domains. Its not worth the extra setup for a secondary then.
Make sure you do not allow recursive queries to the rest of the world (Bind) and make sure you turn on the rate limiting to be safe. As a first step that will really help. Obviously longer term you want move the nameservers away from your current provider and either outsource the management or set up the primary yourself. The general rule we recommend is at least 3 nameservers preferably on multiple continents.
This setup is very robust and will stop everything short of a serious DDOS attack. If that's a real concern then you need to outsource to a specialist. I like and have used netnod.se and Packet Clearing house (https://www.pch.net/) but they are very much not free.
If you are going to do all the DNS yourself then Powerdns is great, but get someone with DNS skills locally to give you real advice.
Re: Ask HN: DNS redundancy, how to do it right?
#28My $dayjob is at a domain registry operator. Find out if your provider will allow you to add your own nameservers and allow zone transfers to them. Most will but find out because you REALLY don't want to synchronize changes manually. You don't need anything fancy, add one or two of your own nameservers. Something at Hetner, OVH, DO, AWS etc are all fine. You only need small basic Linux box with Knot, NSD, or Bind ins…
I gave this a +1, although I have some reservations. As long as they take care of the basics, it's a grind but they can "learn on the job" without putting themselves or others in harm's way.
Being an open recursor isn't as bad as being an open (email) relay. They'll learn about Response Rate Limiting the first time they get used in a reflection attack. And so on.
If they install a web interface to manage DNS all bets are off (what manages the management interface?).
The Basics:
* Having a strong allergic reaction to anyone using the words "simply" or "just".
* Setting up SSH with at least two of a password, a key, IP address restrictions.
* Not allowing dynamic updates until you know what you're doing with security and access controls.
* Keeping patches / software up-to-date.
* Editing zone files / configs by hand.
If they can do those things, then the machine will only be listening on two ports (53 and whatever they put SSH on).
Re: Ask HN: DNS redundancy, how to do it right?
#29Earlier quoted context omitted.
Yep, we also have two. Not a DNS expert but wouldn't half the clients (or their resolvers) stop working if one of your two providers are down? But you can recover by removing the provider that is down from your NS, assuming that the TTL is not too long.
Generally, recursive resolvers will retry queries multiple times and use all the NS records as needed. You shouldn't have a broken NS record if you can avoid it, and it may add a little bit of delay, but things will still work. Some recursive resolvers also keep statistics about authoritative servers, and when a domain had multiple NS records, the recursive resolver will prefer the option that has had the best respon…
> Generally, recursive resolvers will retry queries multiple times and use all the NS records as needed.
IIRC some recursive resolvers ask all of them simultaneously and then just pass the first request received.
Re: Ask HN: DNS redundancy, how to do it right?
#30My $dayjob is at a domain registry operator. Find out if your provider will allow you to add your own nameservers and allow zone transfers to them. Most will but find out because you REALLY don't want to synchronize changes manually. You don't need anything fancy, add one or two of your own nameservers. Something at Hetner, OVH, DO, AWS etc are all fine. You only need small basic Linux box with Knot, NSD, or Bind ins…
I gave this a +1, although I have some reservations. As long as they take care of the basics, it's a grind but they can "learn on the job" without putting themselves or others in harm's way. Being an open recursor isn't as bad as being an open (email) relay. They'll learn about Response Rate Limiting the first time they get used in a reflection attack. And so on. If they install a web interface to manage DNS all bets…
The default for BIND is to only allow recursion on local networks, not just for anyone. I'd be very surprised if other popular DNS software simply openly recurses for anyone.