The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse. The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the…
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever. For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to rol…
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.