Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Problems with low DNS TTLs

00f.net

Post reply on HN
Print view View original

131–140 of 163 posts

Re: Problems with low DNS TTLs

#131

Post by notyourday »

The problem with generalities is that they tend to pick the examples that don't generalize well.

In the case of github's example the author is fixated on DNS where in reality the DNS entry is entry point into fastly's anycast CDN endpoints where the DNS is used to point into the general direction of the correct anycast entrypoint. Fastly's CTO did a great talk a few years ago about load balancing which addressed the DNS issues based on the actual data they have from the edges that service billions of requests.

TL;DR of the DNS portion of that talk is "use as low TTL as you can humanly get away with"

Re: Problems with low DNS TTLs

#132

Post by toast0 »

laurent92 wrote: post #90
rhacker wrote: post #53

Earlier quoted context omitted.

I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly s…

What about 2 minutes during the first hour after a change, 30 minutes during 2 days, 3 hrs after that?

This would be fine for some uses. It's not great for client facing names where you want to be able to react to an incident and have traffic move over quickly.

If it's going to take more than 3 hours to setup the new traffic target anyway, sure. You have time to fiddle with TTLs. If you have servers ready to go elsewhere, 1-5 minute TTLs are nice so you can quickly move things when you notice a problem.

Re: Problems with low DNS TTLs

#133

Post by encoderer »

> The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)

Sure but if it can connect but then pukes out on something like a bad ssl or broken app, it’s not going back and trying another host.

So, when using dns for load balancing, it’s preferable to have a low ttl with a dns record tied to a host health check. If a host goes unhealthy it takes itself out of rotation, auto scaling brings a new one in, and it’s fully warmed up in a minute.

Re: Problems with low DNS TTLs

#134

Post by toast0 »

marcosdumay wrote: post #83
gwittel wrote: post #38

Earlier quoted context omitted.

The major cases revolve around failure recovery, and traffic distribution. A 5 minute outage is not acceptable in many industries or at scale. If a load balancer or DC fails we need to ensure traffic moves away fast. Similarly if you want to take a system out for maintenance or perform migrations.

> A 5 minute outage is not acceptable in many industries or at scale. Well, if that's the case, you better have your redundant systems on your normal DNS entries, because there is no chance you will distribute new entries over the internet in 5 minutes, whatever value you specify at the TTL.

There are tons of things that don't follow TTLs, but a large majority of normal people traffic does.

Easily 90% of new connections will move following the TTL. Of course, some traffic got a DNS result once in 2003 and is going to use that forever. If it's important traffic, you can trace it and follow up with them. If not, you do the best you can and let the rest go.

Re: Problems with low DNS TTLs

#135

Post by Avamander »

intricatedetail wrote: post #121

Author probably never had to switch servers because of failure etc and then had to wait 24 hours until the traffic came back up while losing money and getting angry emails from clients who e.g. bought advertising.

I don't think the author meant 24h TTL should be applied to everything.

Re: Problems with low DNS TTLs

#136

Post by toast0 »

numbsafari wrote: post #62
cortesoft wrote: post #45

Earlier quoted context omitted.

DNS based network load balancing. If you have two data centers, and you want to be able to dynamically and deterministically shift load between them, you want a short TTL so you can control the percentage of traffic going to each data center.

If you want to deterministically shift load, you use routing, not DNS, to manage your load. That’s what is missing from this discussion.

Anycast/BGP traffic engineering is not nearly as accessible as DNS based load balancing. There's several DNS providers you can use to add load balancing and health checks on top of your existing hosting, wherever that is (and you can do it yourself too). Using anycast for this means a specialty hoster, a third party in the data path between your clients and servers, or running your own ASN. I'm sure that gets more precise results than DNS, but it's also a lot more work, and it's harder to replace if the providers involved stop being a good fit.

Determinism isn't necessarily required either. Probabilistic shifting works fine mostly.

Re: Problems with low DNS TTLs

#137

Post by toast0 »

EricE wrote: post #92
aequitas wrote: post #11

Earlier quoted context omitted.

It could help to lower your TTL way before you plan to update. If your current TTL was 24h, you just update it to 5m 24 hours before you plan the actual change of the record itself. The new record can them be set to 24h directly (unless you want a quick turnaround for rollback). It's still no guarantee all changes will propagate within 5 minutes. But it gives some ease of mind to know the bulk of change won't take a…

Yes - it’s called planning. Lowering TTL well in advanced is not a new concept and works very well if you have enough advanced notice from the application owners ahead of time :p

Unfortunately, data centers rarely give 24 hour notice about catastrophic fires or provide advance notice about utility outages when the automatic transfer switch is going to fail.

Re: Problems with low DNS TTLs

#138

Post by pixl97 »

cogman10 wrote: post #116
mlyle wrote: post #95

Earlier quoted context omitted.

Surely we can live with 5-10 minutes for most things, though, right?

I don't see why not. I just don't see there being a huge difference between 1 and 10 minutes in the grand scheme of tech.

That's an order of magnitude which can make a pretty big difference when demand spikes.

Re: Problems with low DNS TTLs

#139

Post by mjevans »

cogman10 wrote: post #91
rhacker wrote: post #53

Earlier quoted context omitted.

I generally agree with this. I think most of us that have set a DNS entry use a low TTL because of experiences we had 5, 10, 15 years ago. Really really bad experiences. I'm not necessarily arguing to increase the TTL, but maybe we should beg the question: with all the new routing tech out there, is low TTL still necessary? K8s Ingress and Cloudfront alone will probably make the customer visible IP addresses nearly s…

I just don't see the harm in a short TTL. Most apps are "bursty" so a 1 minute TTL is more than ample, basically giving them a 1 time penalty on the first request then nothing for the rest of the requests. 1 minute is a long time, you can do a ton of requests in 60,000 milliseconds. On the flip side, setting the TTL long can be a disaster. You can't fix it after the fact. If you have a 1 hour TTL then that's potentia…

It would be nice if failure to connect / TLS handshake failure invalidated the entire related network stack cache for that link. If that were the case I wouldn't mind higher TTLs.

Re: Problems with low DNS TTLs

#140

Post by znpy »

arp242 wrote: post #46
sparrish wrote: post #29

As a sysadmin with 20+ years experience, I've had long TTLs cause issues on several occasions. I've never regretted a short TTL.

The part that's missing in DNS is the ability to tell people that the cache expired. You can't really do this in DNS itself since a cached response will never hit the network. You could add a signal in the application protocol for example: . You'd also have to do that for SMTP, IMAP, etc. Probably not worth the complexity as low TTLs seem to work well enough with few enough downsides. DNS is already a tricky enough p…

Not all the world is HTTP and/or web stuff.

For a lot of things changing domain is quite a hassle, and the old domain will still be around, cached, and considered valid (because its ttl hasn't expired yet).

I agree with GP in saying that a short TTL never created too much issue.

Now, to come to the original issue: short ttl.

What do you expect, in this day and age?

With all this cloud stuff going up and down, being created and destroyed again and again, ephemeral stuff that doesn't even last a whole single day...

Of course you need a short ttl.

Post reply on HN