Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Stop using low DNS TTLs (2019)

#81

Post by throwaway09223 »

jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds"

Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.

DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.

Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.

Re: Stop using low DNS TTLs (2019)

#82

Post by mort96 »

throwaway09223 wrote: post #81
jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for w…

> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.

But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?

Re: Stop using low DNS TTLs (2019)

#83

Post by snowwrestler »

When I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS.

Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before.

Also, DNS service is a lot cheaper than it was years ago, so it doesn’t hurt my budget to send more requests back to the name servers.

Re: Stop using low DNS TTLs (2019)

#84

Post by account42 »

jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.

That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.

Re: Stop using low DNS TTLs (2019)

#85

Post by Serow225 »

mort96 wrote: post #82
throwaway09223 wrote: post #81

Earlier quoted context omitted.

> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds" Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for w…

> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way. But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?

via the magic of Anycast :)

Re: Stop using low DNS TTLs (2019)

#86

Post by account42 »

buzzdenver wrote: post #72
jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconveni…

Not sure about what kind of connection Alaska has but I imagine this is more about reducing latency for new connections than bandwidth.

Re: Stop using low DNS TTLs (2019)

#87

Post by account42 »

snowwrestler wrote: post #83

When I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS. Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before. Also, DNS service is a lot cheaper than it was years…

That sounds like a server setup issue. Nothing about LE prevents you from getting a cert with the old server and moving it to the new server before switchover.

Re: Stop using low DNS TTLs (2019)

#88

Post by manv1 »

jedberg wrote: post #13

DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high. When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut…

If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box.

As the poster says, when it comes to the public internet DNS TTLs are more like guidelines.

Luckily most client DNS caches are cleared on a reboot.

FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".

Re: Stop using low DNS TTLs (2019)

#89

Post by advisedwang »

8organicbits wrote: post #8

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…

If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching)

Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).

Re: Stop using low DNS TTLs (2019)

#90

Post by toast0 »

jedberg wrote: post #32
charcircuit wrote: post #28

Earlier quoted context omitted.

>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching. That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.

I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you t…

You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results.

Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.

Post reply on HN