I thought their DNS was supposed to be rock solid since they use Cloudflare Virtual DNS. Oh well, lesson learned. Back to running my own DNS servers on each droplet, if the DNS is down the droplet is likely down regardless :).
DNS Outage at DigitalOcean
71–80 of 125 posts
Re: DNS Outage at DigitalOcean
#72Does anyone know of a good strategy for DNS failover?
DNS has failover built into the protocol, just have another server listed at your registrar.
Re: DNS Outage at DigitalOcean
#73Their status page at https://status.digitalocean.com is also now giving an intermittent "500 Internal Server Error" nginx error, probably from the load. That's why you should use a service like https://www.statuspage.io for your important stuff, even though creating a status page is a fun side-project for a dev team.
But who monitors the status of statuspage.io?
I know you're joking, but I do: Check out https://StatusGator.com. StatusPage.io has a status page at metastatuspage.com which my company monitors.
Re: DNS Outage at DigitalOcean
#74Yeah this is pretty unfortunate. We have some big investor meetings today and this unfortunately took our marketing site offline. Hopefully they resolve this soon - it's the first time we've ever experienced an issue with their service. We really need fail-overs in place...small team problems.
If you are showing a demo or something, you can still navigate to your DO IP address. Of course, I don't know if other things (images, etc.) on your website also rely on their DNS.
For that kind of emergency, you can point the domain name to the IP address via /etc/hosts (there's a similar file in Windows as well).
Re: DNS Outage at DigitalOcean
#75If you want to get alerted when it comes back up, or you wish you had been alerted when it went down, check out my project: https://StatusGator.com.
StatusGator monitors status pages and sends notifications via email, Slack, and others. You can get alerted to status changes inside Slack and you can ask it the status of a service with a /statuscheck command.
Re: DNS Outage at DigitalOcean
#76DNS is hard. Very hard. It may seems trivial when it works (hint: it's not), but some of the biggest fuck ups I've seen in my professional life were caused by strange DNS things happening or DNS servers going kaboom. I feel the pain of the DO engineers trying to mitigate this issue. I really do.
BS. DNS is a trivial thing to scale, compared to most other web-scale efforts.
Things break when people don't use 20 year old best practices. There is no defense against inexperience and ignorance.
Re: DNS Outage at DigitalOcean
#77Earlier quoted context omitted.
Suppose you have multiple providers, but one of them screws up and authoritatively denies the existence of all of your hosts?
That's what you keep an extremely low ttl for.
Which doesn't mean much when a nontrivial amount of ISPs out there don't respect the TTL settings.
Source: Days-long service degradation caused by customer ISP's caching bad DNS information well beyond the 10 minute TTL we had set.
Re: DNS Outage at DigitalOcean
#78Earlier quoted context omitted.
It's not hard, the problem is everything relies on DNS so when DNS goes down or has problems you have cascading failure.
That's why you use multiple providers.
Unless you use a better resolver than the standard glibc resolver on Linux (e.g. dnsmasq, bind or similar running locally and pointing resolv.conf at it), you appear doomed to slow lookups etc. if your first resolv.conf entry fails, as most of the resolv.conf options that might have helped (if you'd set them) simply don't work or doesn't do anything particularly useful in the versions used in the Linux distro's I've tested it on.
Re: DNS Outage at DigitalOcean
#79People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...
I can't disagree with what you're saying, but I think we are all guilty of this. We expect more out of big name services than might be reasonable. (100% uptime)
How many of us here have failover email services in case Gmail goes down? I think many companies would say they'd lose thousands in productivity if Google Apps suffers an outage yet I'd hazard that very few have failover plans.
Re: DNS Outage at DigitalOcean
#80DNS is hard. Very hard. It may seems trivial when it works (hint: it's not), but some of the biggest fuck ups I've seen in my professional life were caused by strange DNS things happening or DNS servers going kaboom. I feel the pain of the DO engineers trying to mitigate this issue. I really do.
BS. DNS is a trivial thing to scale, compared to most other web-scale efforts. Things break when people don't use 20 year old best practices. There is no defense against inexperience and ignorance.
I took the OPs comment as "it's hard to understand DNS and biggest fuck ups happen because people think they understand DNS when they actually don't".
The problem with DNS is that it can work even when it is configured incorrectly. This makes people who has no idea what they are doing that they actually understand it. The strange issues with DNS only happen with strange configurations. When you follow best practices everything is predictable.