Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Stop using low DNS TTLs (2019)

#91

Post by LinuxBender »

dspillett wrote: post #68
LinuxBender wrote: post #63

Earlier quoted context omitted.

I wonder what those clients are Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.

> Anyone running Unbound DNS can also do this Yep. Just checked the docs for that and “cache-min-ttl” is a thing. For those unfamiliar, unbound is a pretty common resolver: it is the default for most BSDs and things based on them (such as pfSense) and used by the popular PiHole and its forks. How common it is to use this setting I can't comment on. > Some versions and distributions of Java do all manor of odd things…

The reason I mention Java is that when updating DNS in a business, there may be business to business flows that are critical. This is becoming more common than ever with cloud-to-cloud-to-cloud dependencies. Mishaps in cloud services can affect a service used by a large number of people. That is why I always tried to teach people to design apps so that TTL could in theory be 1 week and it would not matter. e.g. apex domain anycast IP used for authentication, then direct browsers, API clients and cloud daemons to a service dedicated domain name that is also anycast and uses health checks behind the scenes to take nodes out the picture rather than relying on DNS.

Re: Stop using low DNS TTLs (2019)

#93

Post by WirelessGigabit »

snowwrestler wrote: post #83

When I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS. Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before. Also, DNS service is a lot cheaper than it was years…

Use acme.sh, then all of this happens magically. Also, look at ZeroSSL.

Re: Stop using low DNS TTLs (2019)

#94

Post by remram »

technovader wrote: post #78

There are use cases for low TTLs and use cases for high TTLs The statement here is misguided. Use whatever TTL serves your purpose. DNS was designed to be distributed and scalable. And in my opinion it is THE MOST scalable protocol as it basically allowed the entire internet to work that way. Any user in the world can get any public DNS record for a site.

Now if only they could have put the components in the right order...

Re: Stop using low DNS TTLs (2019)

#95

Post by ikiris »

cmeacham98 wrote: post #40
ikiris wrote: post #23

Earlier quoted context omitted.

And yet most of that doesn't matter, because you almost always have local resolvers that do cache properly.

Your average consumer router is a pile of garbage that barely works. I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it. Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.

The latency to most ISP cores is low ms range. You really should just stop digging.

Re: Stop using low DNS TTLs (2019)

#96

Post by throwaway09223 »

mort96 wrote: post #82
throwaway09223 wrote: post #81

Earlier quoted context omitted.

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for w…

> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?

You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example.

DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.

Re: Stop using low DNS TTLs (2019)

#97

Post by silisili »

mort96 wrote: post #82
throwaway09223 wrote: post #81

Earlier quoted context omitted.

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for w…

> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?

Yes, with BGP and anycast.

What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.

Re: Stop using low DNS TTLs (2019)

#98

Post by russellendicott »

jonhohle wrote: post #73
jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future. 0 - https://docs.oracle.com/javase/7/docs/techno…

I seem to recall that HAProxy and Squid used to do this too.

Re: Stop using low DNS TTLs (2019)

#99

Post by akira2501 »

withinboredom wrote: post #65
mrb wrote: post #29

Earlier quoted context omitted.

If your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing. Unless your traffic is non-revenue generating or somehow poorly monetizable...

> So a few thousand dollars is nothing. In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.

> a few thousands in UNEXPECTED costs, is huge.

It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month?

It's a fixed _overhead_ issue, not a fixed _budget_ issue.

Re: Stop using low DNS TTLs (2019)

#100

Post by M3L0NM4N »

advisedwang wrote: post #89
8organicbits wrote: post #8

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…

If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching) Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rath…

I think the amount of people that would see the error either way would be the same, assuming constant fix time and traffic. This means lowering the TTL as much as possible should be the right solution, because those affected will only have the issue for as long as the fix takes (hopefully less than 1hr) whereas if it is kept at 1hr those affected by the error will have it for an entire hour.

Disclaimer I'm not OP

Post reply on HN