Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Stop using low DNS TTLs (2019)

#21

Post by donmcronald »

8organicbits wrote: post #8

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…

> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.

This is my preferred way of doing it, especially for MX records.

Re: Stop using low DNS TTLs (2019)

#22

Post by musicale »

jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)

Re: Stop using low DNS TTLs (2019)

#23

Post by ikiris »

cmeacham98 wrote: post #11

Most clients ignore TTLs anyways, so your 1 day TTL barely is helping you're website's performance at all: https://www.ctrl.blog/entry/dns-client-ttl.html

And yet most of that doesn't matter, because you almost always have local resolvers that do cache properly.

Re: Stop using low DNS TTLs (2019)

#24

Post by outofmyshed »

Anyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with.

I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.

Re: Stop using low DNS TTLs (2019)

#26

Post by occamrazor »

jiggawatts wrote: post #14
jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like…

Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?

Re: Stop using low DNS TTLs (2019)

#27

Post by sofixa »

blueflow wrote: post #5
antonvs wrote: post #3

> DNS caching has become next to useless. Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.

It made as much sense as it does now, but differently from back then, people today neglect the performance and reliability impact it has.

Not really, because infrastructure is much more dynamic today. Maybe if your IPs never changed long TTLs are OK, but when you're running behind a cloud load balancer, with changing IPs and failover to different regions/providers, low TTLs make more sense.

Re: Stop using low DNS TTLs (2019)

#28

Post by charcircuit »

jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.

That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.

Re: Stop using low DNS TTLs (2019)

#29

Post by mrb »

outofmyshed wrote: post #24

Anyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with. I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.

If your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing.

Unless your traffic is non-revenue generating or somehow poorly monetizable...

Re: Stop using low DNS TTLs (2019)

#30

Post by darkr »

occamrazor wrote: post #26
jiggawatts wrote: post #14

Earlier quoted context omitted.

We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like…

Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?

It varies; for direct peering/connections where a link goes down (and fast failover protocol is used), usually 5 seconds or so. Global propagation takes in the order of 5 minutes or less: https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-updat...
Post reply on HN