Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Problems with low DNS TTLs

00f.net

Post reply on HN
Print view View original

71–80 of 163 posts

Re: Problems with low DNS TTLs

#71

Post by xoa »

cookiengineer wrote: post #47
jabart wrote: post #37

Earlier quoted context omitted.

Same, saw a DNS entry for an internal DB endpoint get updated with an 8 hour TTL for a planned failover. DB Admins went to sleep as everything was fine, everyone woke up 8 hours later with everything failing to connect. Had to flush the DNS on each internal server to fix that issue.

The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.

I'd kind of like if there was some ability to have more complex multi-part TTLs as an option along with a default TTL (the current one). So I could specify

  Default TTL: 12 hours;
  [ to ] TTL: 10 minutes;
or even

  Default TTL: 12 hours;
  [Thu 0000-1200, repeating]: 5 minutes;
  [ to ] TTL: 10 minutes;
So with no further effort all downstream caches/clients can basically have advanced notice of regular maintenance windows as well as planned maintenance and just automatically adapt. Of course this wouldn't deal with true emergencies, but it might lower the overhead for a huge amount of regular stuff that otherwise tempts people to set it low and leave it that way.

I dunno, I'm sure there's other downsides I haven't thought of, and proper implementation would require thinking through side effects. But after a long time dealing with it feels like there's some room for something beyond one single TTL ever which must be specifically changed (with a wait for propagation) well ahead of time whenever anything planned with a risk of issues needs to be done. Maybe?

Re: Problems with low DNS TTLs

#73

Post by axaxs »

jamescun wrote: post #33
sparrish wrote: post #29

As a sysadmin with 20+ years experience, I've had long TTLs cause issues on several occasions. I've never regretted a short TTL.

This mirrors my experience. I've seen more problems caused by the JVM, by default on some configurations, caching DNS indefinitely, regardless of TTL, than caused by a short TTL.

Is that fixed yet?

I remember having to bounce Java apps every time DNS changed, which never made sense to me. It's literally the point of DNS to not have to do that.

Re: Problems with low DNS TTLs

#74

Post by icedchai »

axaxs wrote: post #73
jamescun wrote: post #33

Earlier quoted context omitted.

This mirrors my experience. I've seen more problems caused by the JVM, by default on some configurations, caching DNS indefinitely, regardless of TTL, than caused by a short TTL.

Is that fixed yet? I remember having to bounce Java apps every time DNS changed, which never made sense to me. It's literally the point of DNS to not have to do that.

It's fixed in newer JVMs ("newer" meaning anything in the past 10 years.)

Re: Problems with low DNS TTLs

#75

Post by Diggsey »

xoa wrote: post #71
cookiengineer wrote: post #47

Earlier quoted context omitted.

The issue here is whether a one-time migration lazyness justifies millions or billions of DNS requests to a web service that could've been saved for customers. And I do not think it does. DNS without caching is useless traffic overhead. Just like HTTP responses without gzip compression. DNS entries almost never change, therefore it should be cached accordingly.

I'd kind of like if there was some ability to have more complex multi-part TTLs as an option along with a default TTL (the current one). So I could specify Default TTL: 12 hours; [ to ] TTL: 10 minutes; or even Default TTL: 12 hours; [Thu 0000-1200, repeating]: 5 minutes; [ to ] TTL: 10 minutes; So with no further effort all downstream caches/clients can basically have advanced notice of regular maintenance windows a…

I don't think you'd need to further complicate DNS, just have a service running that checks a calendar and syncs your DNS TTLs appropriately.

eg.

- Check if there's a maintenance window in the next [max TTL]

- If not, set TTL = [max TTL]

- Otherwise, set TTL = [time until maintenance window]

- Repeat

Re: Problems with low DNS TTLs

#76

Post by AtNightWeCode »

One purpose for a low TTL in the solutions I have built is that you want to change the IP. So first you hit the DNS. You get an IP from some main location. Then after the first request you figure out where the user is located. Perhaps spins up some container close to the user. Then on consecutive requests you get an IP much closer to the user.

Another usage is to load balance out a lot of users to different web nodes for instance.

Edit: spelling

Re: Problems with low DNS TTLs

#77

Post by xbar »

Customer-facing DNS should have TTLs on the order of 15 to 30 minutes. Halving those values to estimate TTL value to the end user, you get 7 to 15 minutes of cached DNS. That's about right for most user interactions on the web.

Much longer and you run into all the trouble that operators have with keeping DNS accurate. DNS is hard. It is easy to break. And 15 to 30 minutes of waiting is about as much normal human attention span you can apply to a problem that sounds like, "Ok, we're all done, is DNS ok?"

5 to 10 minute TTLs only benefit operators. Certainly, any TTL less than 5 minutes is an indicator that your operators have no faith whatsoever in their ability to manage DNS.

Re: Problems with low DNS TTLs

#78

Post by avidiax »

The problem is that the DNS TTL is a feature designed for a static internet of the 70's or 80's.

What this points to is a need for an authenticated DNS pushes for refresh/invalidation.

All supporting resolvers could keep a list of supporting clients that were told that "foo is at address 42". If the record changes, the authoritative DNS server sends a DNSSEC signed unsolicited response to all previous requesters to update their records. Obviously the TTL can be extended to keep the cache of requestor IPs reasonably sized.

Will this happen? Well, for UDP DNS it depends on DNSSEC, which is already not well supported, and it fixes something that is broken but not terribly so. One could imagine Google arranging this between its DNS resolvers and Chrome, for instance.

For DNS over HTTPS, this becomes much more feasible.

Re: Problems with low DNS TTLs

#79

Post by gertrunde »

samvher wrote: post #49
JimWestergren wrote: post #20

Earlier quoted context omitted.

But 5 minutes should be fine? It surprised me that so many has 1 second or 20 second TTL.

If I need to get a web service up and I can save 4 minutes by setting a low TTL when I configure my DNS record why wouldn't I?

Because you're pushing the cost on to someone else.

If your DNS hosting provider charged you per query (some do, especially when adding features like health checks & load balancing), then it might make a big difference.

Re: Problems with low DNS TTLs

#80

Post by yjftsjthsd-h »

stevekemp wrote: post #65
yjftsjthsd-h wrote: post #28

Earlier quoted context omitted.

> But I bet you still ran forwarders on the old hosts for at least an hour after you cut over DNS. I promise you we did not.

That's a surprise - I've handled migrations like this in the past, and we always setup a simple proxy to forward traffic for a while. I've definitely lost count of the number of clients that would cache the old IPs, despite valid and low TTLs being in-place well in advance of a migration.

The impression I got from the senior sysadmins was that we considered clients caching records beyond TTL to be a bug on their side and not our problem, and (importantly) the nature of our business/clients allowed us to make that determination and not take corrective measures to compensate for client-side misconfigurations. As such, practicing traffic would have been considered at best unnecessary work (and at worst comprising our testing process and encouraging bad behavior).
Post reply on HN