Ask HN: DNS redundancy, how to do it right?

Are you getting "DDoS'ed" from the DNS Server or from the HTTP Server?

Usually the later is more common since DNS tends to be a quite robust software to handle some levels of heavy the traffic (and the protocol being _lighter_ than HTTP).

Anyway you should separate the DNS Server from the Web Server, for this particular case personally i recommend "DNS Cloud providers" like AWS Route 53, they give you by default 4 different geo-located points and provides an API if you want to fight back the DDoS'ers (by changing your DNS records to 127.0.0.1 or 255.255.255.255 for a short period of time), usually these solve the email issue.

As for the web server, this is tough a well placed DDoS won't be stopped even Cloudflare have been hit with huge attacks that they couldn't handle (despite what their PR department says, the fact that even a bigger network than Cloudflare like Akamai couldn't protect Brian Kerbs tells you a lot about how tough these space is), best way is null routing the bad actors and spreading different ways to access your services like asking customers to go to frontendXYZ.mydomain.tld.

At $oldjob, we once got an extortion email of dubious credibility from an unidentifiable party, claiming they would DDoS our (a lucrative commercial web enterprise with lots of daily users) infrastructure, if we refused to pay 100 BTC (the BTCUSD rate was a lot lower back then than what it is now ;)). As the infrastructure lead, I used this as an opportunity (albeit in a bit of a hurry ;)) to strengthen our resilience against this kind of threat, also on the DNS level.

The pair of authoritative nameservers, which we were self-hosting at the colo space we rented, was based on PowerDNS, with a replicated PostgreSQL database behind it. A "shadow master" postgresql instance was were control over zone data could be exercised, and that postgres instance used streaming replication to shuttle its dataset to read-only secondaries over a purpose-specific SSH tunnel (nowadays, we'd probably be using wireguard instead). There, powerdns authoritative DNS server instances picked up the zone data from the host-local postgres databases, and served that up by means of DNS.

This setup proved very easily extensible (spinning up a new, additional secondary was a matter of a few minutes via a simple ansible playbook that set up a new SSH tunnel, a postgres hot standby, and a new pdns instance that drew its zone data from the local postgres instance), and we chose to deploy two additional nameservers at dedicated server providers in nearby Europe to host our tertiary and quaternary authoritative DNS servers. The only remaining, but tedious task left was updating all the glue records for the domains we handled on these nameservers.

In the end, the entire threat proved hollow, as the deadline passed with zero impact on any of our infra. We never learnt if it was just empty to begin with, or if the adversary decided not to bother attacking a visibly well-prepared site. But the resilience-improved DNS infrastructure was a nice thing to be able to rely on in the coming years, and I think pretty much the same architecture/setup is still in operation to this day.

You can have multiple secondary servers, what is not a good idea is to have just one, because whatever it happens with it things may fail elsewhere (i.e. mails lost, that may be worse than losing connectivity with your website for a while). And once the zone is setup and the secondary working, you only need to modify things in the primary.

Regarding TTLs, how frequently you modify those records, or how probable is that you have to modify them in a near future? Those times will say to resolvers for much time they could cache that information, and give a time for propagation of a change in that information. With current bandwidths and having slaves you can have relatively frequent updates, but have some margin to have meaningful caches in remote resolvers to speed up access to your sites.

Pay someone.

But if you really want. PowerDNS + MySQL replication for the "slaves" then put slaves in few different places with good bandwidth.

Instant updates, very bulletproof.

The general advice is that you should have more than 1 name server and they should be on different networks/servers, to prevent the exact issue you're suffering, that a DDoS on one network doesn't cause an entire namespace outage. You are right to think about increasing the TTL of anything that's static and constant, like MX records. That TTL value buys time when there are issues before it is propagated to hosts that need to query your records as it's not cached at their end. Another trick, is to have the authoritative server where you perform the updates not referenced in your domains NS records, and instead only list the secondary (replica) servers in your NS records. So you maintain control and any attack based on NS records is on the replicas, and they refetch the record from the authoritative server periodically (based on SOA record settings).