Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

Post reply on HN
Print view View original

61–70 of 110 posts

Re: Stop using low DNS TTLs (2019)

#61

Post by wahern »

bagels wrote: post #42

Is there a push mechanism in DNS? My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs. (I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)

There's DNS NOTIFY (RFC 1996), which is commonly used to trigger a pull. But to minimize malfeasance it's usually ignored unless authenticated (e.g. via DNS TSIG) or appears to originate from a known address. The typical scenario is a master nameserver notifying a list of [known] slaves about a zone update.

There's also DNS Push (RFC 8765), but it's a very recent mechanism. I'm not familiar, but I doubt it's widely supported.

Re: Stop using low DNS TTLs (2019)

#62

Post by jve »

donmcronald wrote: post #21
8organicbits wrote: post #8

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…

> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. This is my preferred way of doing it, especially for…

Or you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.

Re: Stop using low DNS TTLs (2019)

#63

Post by LinuxBender »

dspillett wrote: post #41
Tepix wrote: post #35

Earlier quoted context omitted.

Quoting from https://cr.yp.to/djbdns/tinydns-data.html : Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.

I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem. I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs ( My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short…

I wonder what those clients are

Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.

Re: Stop using low DNS TTLs (2019)

#64

Post by nijave »

It seems the only reason given is "some clients are broken"?

They mention latency but I wonder what the real world impact is. On a home network, most users are using ISP supplied DNS which /usually/ has pretty low latency and will have a huge cache already built up. Corporate users will have DNS cached to all the common host they're visiting.

Anecdotally, AWS ALBs (HTTP load balancers) have a CNAME with a 10 minute TTL (maybe it is 60 seconds) and can have their IPs change 100s of times a month and this works fine as a default configuration for many use cases.

Re: Stop using low DNS TTLs (2019)

#65

Post by withinboredom »

mrb wrote: post #29
outofmyshed wrote: post #24

Anyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with. I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.

If your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing. Unless your traffic is non-revenue generating or somehow poorly monetizable...

> So a few thousand dollars is nothing.

In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.

Re: Stop using low DNS TTLs (2019)

#66

Post by ilyt »

jiggawatts wrote: post #14
jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

We should do the right thing even if other people do the wrong thing. Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit. Fundamentally however, giving clients control over when failover occurs means giving up control over failover. Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft. The conversation went like…

> Fundamentally however, giving clients control over when failover occurs means giving up control over failover.

Fundamentally the problem is that you want to direct a client to particular datacenter and it is not always shortest part (shortest != lowest latency) so anycast DNS isn't helping here.

There is reason biggest CDNs use it to direct to particular DC instead of " just anycasting everything" like you're proposing, it's suboptimal.

> You're all probably familiar with 8.8.8.8 and 1.1.1.1,

And both companies behind that still direct you to particular datacenter, not anycasted IP, for the actual content.

Anycast IP are also "expensive", you need to use entire /24 route to get one, and it takes routing table space in every router's memory that is limited, and requires costly replacement if it runs out everywhere, not just where you distribute it.

Re: Stop using low DNS TTLs (2019)

#67

Post by nijave »

dspillett wrote: post #41
Tepix wrote: post #35

Earlier quoted context omitted.

Quoting from https://cr.yp.to/djbdns/tinydns-data.html : Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.

I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem. I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs ( My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short…

HughesNet used to run a caching DNS resolver on their customer equipment circa 2016 (not sure if they still do) which overwrote TTLs to 12 hours.

Java/JVM historically ignored TTL and just did a single DNS lookup and cached the result unless configured otherwise.

PGBouncer ignores DNS TTL and has its own configuration.

I've seen lots of software that will create a "connection" object and re-use that with TCP reconnect handling. In practice, that means the DNS resolution happens when the connection is recreated but not necessarily when the TCP connection reconnects due to timeout/failure. I've also seen some issues with long lived connections (where the connection greatly outlives the TTL). Long lived connections aren't necessarily a DNS TTL problem but they mean DNS load balancing/redirection don't always work how you expect.

That said, the solution is usually to just fix the issue in the software and not ignore DNS TTLs.

As counter examples, AWS ALBs and Kubernetes service discovery both rely on short TTLs with fairly good success.

Re: Stop using low DNS TTLs (2019)

#68

Post by dspillett »

LinuxBender wrote: post #63
dspillett wrote: post #41

Earlier quoted context omitted.

I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem. I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs ( My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short…

I wonder what those clients are Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.

> Anyone running Unbound DNS can also do this

Yep. Just checked the docs for that and “cache-min-ttl” is a thing. For those unfamiliar, unbound is a pretty common resolver: it is the default for most BSDs and things based on them (such as pfSense) and used by the popular PiHole and its forks. How common it is to use this setting I can't comment on.

> Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's

This at least is less of an issue if all you are futzing with the DNS for is services intended to be consumed by a web browser: if common browsers and DNS resolvers behave OK then you are good, if someone consuming your stuff through their own code hits a problem they can work around it themselves :)

It goes without saying that only a fool would play trick like this for email as that is already a mountain of things that can be easily upset by anything off the beaten track.

EDIT:

Also, yep, people are using the setting in their normal environments: https://stackoverflow.com/questions/21799834/how-to-determin... (that person currently having it set to 30 minutes)

Re: Stop using low DNS TTLs (2019)

#69

Post by eatbitseveryday »

jve wrote: post #62
donmcronald wrote: post #21

Earlier quoted context omitted.

> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address. This is my preferred way of doing it, especially for…

Or you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.

That’s exactly what was said in the above comments

Re: Stop using low DNS TTLs (2019)

#70

Post by megiddo »

IME DNS TTLs are low because debugging DNS during development is a pain.

So you set them low the first time you run into a DNS issue where you had to wait an hour or a day to resolve some bad configuration and your "fuck this" meter went from 0 to 100.

Post reply on HN