Another reason to set a low DNS TTL, is if you are running a service with very limited resources and still want some kind of HA / Disaster recovery. For instance you setup your web app in both Digital Ocean and Linode, with a system that takes the database (for instance mysql replication) and the web app synchronized in the "replica" VM. When something bad happens and you can't recover, you switch DNS and that's it.
Stop using low DNS TTLs (2019)
51–60 of 110 posts
Re: Stop using low DNS TTLs (2019)
#52DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…
When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them?
From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.
Re: Stop using low DNS TTLs (2019)
#53Earlier quoted context omitted.
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you t…
I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out.
People without cached records will always get the returned servers with the lowest load.
I was referring to load balancing and not splitting traffic for an experiment.
Re: Stop using low DNS TTLs (2019)
#54Earlier quoted context omitted.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you t…
I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.
Re: Stop using low DNS TTLs (2019)
#55Fascinating read. Of course I'm no big business but for the last 20 years I've done it like this for non-critical stuff:
- NS Records at 86400
- A and CNAME at 3600
- 1-2 days before moving stuff to other infra (ISP, DNS reseller, etc), switch to 600 or lower, then 1-2 days later switch back.
And every single time there was some problem, setting the TTL to < 1minute would not have helped, so I guess I'll stick to it.
Re: Stop using low DNS TTLs (2019)
#56DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…
> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.
DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.
Re: Stop using low DNS TTLs (2019)
#57The way of measuring this affects (and skews) the outcome. DNS queries with low TTLs are requested more frequently and because of that, the author seeing more of them pass through his patched DNS relay, which was only left running for a few hours instead of for at least the max TTL they wanted to measure.
That's what I thought of as well. Data can be presented in many ways to show many meanings at first sight. If charts would be generated in number of unique NAMEs in relation to TTL, readers would probably get a diffetent feeling. Or maybe not. I don't know, but this issue probably has a name ... selection bias?
Yeah,that was one of my first thoughts as well. I guess the raw.github example gets not only more traffic (irrelevant here) but probably also many more requests than hundreds of other sites combined.
Also no clue how actual RR DNS load balancing (without anycast) is still widely practiced, because I guess that also influences it a lot, you may be able to fix a DC going down on a completely different level than DNS. And my experience with cheap home routers is basically that they might cache hostnames for a month (we had stuff with tracking pixels in 2013-2017, EOLing some stuff was wild, we switched DNS and days and weeksl later still got valid requests to the old IPs).
Re: Stop using low DNS TTLs (2019)
#58Earlier quoted context omitted.
We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like…
Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?
They are added and withdrawn on demand, or dropped all at once if router loses connectivity with another.
So really it's processing time of every node in the chain. Few seconds to few minutes, not really that different from DNS.
The advantage is that there is no updates if there is no changes, instead of update every time TTL expires but that's about it.
Re: Stop using low DNS TTLs (2019)
#59Earlier quoted context omitted.
We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like…
Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?
They aren't really cached. Routes are kept in the routing table as long as peers are responsive, if they stop responding to hello's or a link goes down to directly connected peer the routes are removed (or replaced with a less preferred one). While its possible to configure a router to keep routes in the table for a peer that is down for some time its typically not used unless you have a good reason for it.
Re: Stop using low DNS TTLs (2019)
#60Is there a push mechanism in DNS? My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs. (I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)
There is the NOTIFY extension, but it's for master servers to notify secondary servers that they should do a pull. Not really usable for unknown general clients, of course.