Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop using low DNS TTLs (2019)

blog.apnic.net

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Stop using low DNS TTLs (2019)

#101

Post by withinboredom »

akira2501 wrote: post #99
withinboredom wrote: post #65

Earlier quoted context omitted.

> So a few thousand dollars is nothing. In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.

> a few thousands in UNEXPECTED costs, is huge. It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month? It's a fixed _overhead_ issue, not a fixed _budget_ issue.

You should be reserving those instances so it IS a FIXed cost, with very few exceptions.

Re: Stop using low DNS TTLs (2019)

#102

Post by charcircuit »

toast0 wrote: post #90
jedberg wrote: post #32

Earlier quoted context omitted.

I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data. Why would you t…

You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results. Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "clo…

If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load.

Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.

Re: Stop using low DNS TTLs (2019)

#103

Post by cmeacham98 »

ikiris wrote: post #95
cmeacham98 wrote: post #40

Earlier quoted context omitted.

Your average consumer router is a pile of garbage that barely works. I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it. Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.

The latency to most ISP cores is low ms range. You really should just stop digging.

Maybe you live in a country where ISP routers are well designed and low latency, but where I live in the US the primary latency in DNS lookup is almost always the round trip to the horribly configured and slow ISP DNS.

Re: Stop using low DNS TTLs (2019)

#104

Post by toast0 »

charcircuit wrote: post #102
toast0 wrote: post #90

Earlier quoted context omitted.

You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results. Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "clo…

If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load. Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.

You're suggesting that the DNS server(s) get realtime load feedback and adjust appropriately. That's possible, but not always available.

Re: Stop using low DNS TTLs (2019)

#105

Post by akira2501 »

withinboredom wrote: post #101
akira2501 wrote: post #99

Earlier quoted context omitted.

> a few thousands in UNEXPECTED costs, is huge. It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month? It's a fixed _overhead_ issue, not a fixed _budget_ issue.

You should be reserving those instances so it IS a FIXed cost, with very few exceptions.

There are many AWS services for which you cannot make reservations. You can get better per unit rates by pre-purchasing, but even with instances, network transfer is a per unit factor.

Your only way out is the built in tools to limit service in these cases or to build your own circuit breakers and implement them, or make your usage so small as to not truly require the cloud in the first place.

Re: Stop using low DNS TTLs (2019)

#106

Post by charcircuit »

toast0 wrote: post #104
charcircuit wrote: post #102

Earlier quoted context omitted.

If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load. Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.

You're suggesting that the DNS server(s) get realtime load feedback and adjust appropriately. That's possible, but not always available.

Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet.

>That's possible, but not always available.

If you build it, then it's always available.

Re: Stop using low DNS TTLs (2019)

#107

Post by toast0 »

charcircuit wrote: post #106
toast0 wrote: post #104

Earlier quoted context omitted.

You're suggesting that the DNS server(s) get realtime load feedback and adjust appropriately. That's possible, but not always available.

Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet. >That's possible, but not always available. If you build it, then it's always available.

Sure, but a lot of people use other people's DNS services. The number of times I've been allowed to build a DNS service for my employer is zero. I've used static DNS, where you just put in your X number of A records and hope; dynamic services where you put in X and have them serve only Y of them in any request and hope. I've used services with load feedback, but those are always a much higher tier.

Re: Stop using low DNS TTLs (2019)

#108

Post by charcircuit »

toast0 wrote: post #107
charcircuit wrote: post #106

Earlier quoted context omitted.

Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet. >That's possible, but not always available. If you build it, then it's always available.

Sure, but a lot of people use other people's DNS services. The number of times I've been allowed to build a DNS service for my employer is zero. I've used static DNS, where you just put in your X number of A records and hope; dynamic services where you put in X and have them serve only Y of them in any request and hope. I've used services with load feedback, but those are always a much higher tier.

>Sure, but a lot of people use other people's DNS services

There are many of DNS services which offer an API for querying and updating records.

It is very easy to write a service / script that just fines the X least loaded servers and then call an API to set those as the available records. In practice you will also want to implement some monitoring that it is actually working.

Re: Stop using low DNS TTLs (2019)

#109

Post by 8organicbits »

advisedwang wrote: post #89
8organicbits wrote: post #8

The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance. I've seen this happen on internal networks where IT support tries to guide end users throu…

If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching) Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rath…

Testing strategy is also pretty important here. If you can run integration tests that bypass DNS caches immediately after the change, then you should know quick if things are working. If both old and new DNS settings are valid, the slow onset works great and minimizes impact of an error.

I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.

Re: Stop using low DNS TTLs (2019)

#110

Post by dexterdog »

solatic wrote: post #17
antonvs wrote: post #3

> DNS caching has become next to useless. Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.

AWS Route53 charges per-query. At scale, using longer TTLs, if clients respect the longer TTLs, has a measurable impact on your bill.

That's why you use aliases as much as possible since they are not charged.
Post reply on HN