Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

11–20 of 110 posts

Re: Stop using low DNS TTLs (2019)

#12
I like changing my IP address from time to time. I host my own VPN and connect to it through a domain. When I update my IP, I have no internet access for whatever the TTL length is.

Re: Stop using low DNS TTLs (2019)

#13
DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.

When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.

And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48.

It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice.

[0] https://www.ctrl.blog/entry/dns-client-ttl.html

Re: Stop using low DNS TTLs (2019)

#14
post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

We should do the right thing even if other people do the wrong thing.

Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit.

Fundamentally however, giving clients control over when failover occurs means giving up control over failover.

Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft.

The conversation went like this:

> "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem.

> "Client software often ignores DNS changes."

> "We designed Azure so that Traffic Manager does failover and load balancers are always active-active."

> "That's nice, but client software ignores DNS changes."

> "Umm..."

Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible.

The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-...

Re: Stop using low DNS TTLs (2019)

#16
post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not.

If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?

Re: Stop using low DNS TTLs (2019)

#17
post #3

> DNS caching has become next to useless. Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.

AWS Route53 charges per-query. At scale, using longer TTLs, if clients respect the longer TTLs, has a measurable impact on your bill.

Re: Stop using low DNS TTLs (2019)

#18
post #10

The way of measuring this affects (and skews) the outcome. DNS queries with low TTLs are requested more frequently and because of that, the author seeing more of them pass through his patched DNS relay, which was only left running for a few hours instead of for at least the max TTL they wanted to measure.

That's what I thought of as well. Data can be presented in many ways to show many meanings at first sight. If charts would be generated in number of unique NAMEs in relation to TTL, readers would probably get a diffetent feeling. Or maybe not.

I don't know, but this issue probably has a name ... selection bias?

Re: Stop using low DNS TTLs (2019)

#19
post #16
post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not. If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too f…

[deleted]

Re: Stop using low DNS TTLs (2019)

#20
post #3

> DNS caching has become next to useless. Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.

It makes a lot of sense, if we think about how the protocol works. Without caching, I assume every client would send every query in it's entirety to the root server, right?

Since a single zone can have multiple labels ({abc.def}.ghi.example - abc.def is a NAME in ghi.example), the server needs to receive the entire requested name in order to properly respond (yeah, root queries usualy optimize this, because all NAMEs there are one label long).

Post reply on HN