I'm using Netlify DNS for most of my websites. Usually I set TTL low (~600 seconds) in order to quickly change configuration in case I fuck up. When things are working though, I remember that Netlify doesn't have an edit function for DNS, so I just leave it as is.
Stop using low DNS TTLs (2019)
31–40 of 110 posts
Re: Stop using low DNS TTLs (2019)
#32DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data.
Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
Re: Stop using low DNS TTLs (2019)
#33DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…
I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not. If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too f…
That's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.
Re: Stop using low DNS TTLs (2019)
#34One cool thing about DJBs tinydns is that you can configure TTLs if you plan to roll over to another system at a particular point in time.
I.e. you can have multiple records each with its own time range where it is valid.
From the tinydns-data documentation:
--
You may include a timestamp on each line. If ttl is nonzero (or omitted), the timestamp is a starting time for the information in the line; the line will be ignored before that time. If ttl is zero, the timestamp is an ending time (``time to die'') for the information in the line; tinydns dynamically adjusts ttl so that the line's DNS records are not cached for more than a few seconds past the ending time. A timestamp is an external TAI64 timestamp, printed as 16 lowercase hexadecimal characters. For example, the lines
+www.heaven.af.mil:1.2.3.4:0:4000000038af1379
+www.heaven.af.mil:1.2.3.7::4000000038af1379
--
Do other authorative resolvers support this, too?
Re: Stop using low DNS TTLs (2019)
#35The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…
Quoting from https://cr.yp.to/djbdns/tinydns-data.html :
Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.
Re: Stop using low DNS TTLs (2019)
#36An outage at a large (but not mega) company will likely have a 5-10 minute period to get looked at, a 5-10 minute period to understand the problem, and a 5-10 minute period for resolution. Any DNS related problem with a TTL higher than 5-10 minutes could greatly extend an outage.
Re: Stop using low DNS TTLs (2019)
#37Earlier quoted context omitted.
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you t…
I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
Re: Stop using low DNS TTLs (2019)
#38If you use a pihole, you can force the minimum ttl for downstream clients (upto a maximum value of 3600s) by setting, for example, min-cache-ttl=2400 in /etc/dnsmasq.d/ Never had a problem with this.
Of course, you can force a flush of the cache using `pihole restartdns` in case you need to.
Re: Stop using low DNS TTLs (2019)
#39DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…
[deleted]
Re: Stop using low DNS TTLs (2019)
#40Most clients ignore TTLs anyways, so your 1 day TTL barely is helping you're website's performance at all: https://www.ctrl.blog/entry/dns-client-ttl.html
And yet most of that doesn't matter, because you almost always have local resolvers that do cache properly.
Your average consumer router is a pile of garbage that barely works.
I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it.
Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.