I think the most annoying aspect of this outage are their updates. Three updates and they all say the same thing with no meaningful information as to what's causing this. Likely they may not have much information, but you'd think there would be something more than what they've been posting for the past hour. Good times!
DNS Outage at DigitalOcean
21–30 of 125 posts
Re: DNS Outage at DigitalOcean
#22Their status page at https://status.digitalocean.com is also now giving an intermittent "500 Internal Server Error" nginx error, probably from the load. That's why you should use a service like https://www.statuspage.io for your important stuff, even though creating a status page is a fun side-project for a dev team.
Re: DNS Outage at DigitalOcean
#23This is the first DNS outage I've experienced with them in 3+ years, then again I host everything in their NY regions.
I've never experienced an outage of any kind with DO, so also first time. I also host all my droplets in the NYC regions.
Re: DNS Outage at DigitalOcean
#24Yeah, tried to access our site and it was down. Really was expecting more out of Digital Ocean than to fuck up such an integral part of their infrastructure. In the future we'll be transitioning away from their DNS solution because this is unacceptable.
I hope your clients/users are as understanding and civil as you are.
In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).
Re: DNS Outage at DigitalOcean
#25Does anyone know of a good strategy for DNS failover?
Well, there's a couple of strategies: - IP-diverse nameservers - TLD-diverse nameservers - BGP anycast IP-diverse nameservers requires that you expect that your DNS servers will go down rather than start returning bad results - I highly recommend having some sort of mechanism to hard-terminate access to those machines. TLD-diverse nameservers is just an extra strategy for reducing the risk that an upstream TLD issue…
- Implementation-diverse nameservers
Use multiple implementations, e.g. NSD/BIND for authoritative servers and Unbound/BIND for resolvers, to mitigate against implementation-specific bugs and vulnerabilities.
Re: DNS Outage at DigitalOcean
#26DNS is hard. Very hard. It may seems trivial when it works (hint: it's not), but some of the biggest fuck ups I've seen in my professional life were caused by strange DNS things happening or DNS servers going kaboom. I feel the pain of the DO engineers trying to mitigate this issue. I really do.
> I feel the pain of the DO engineers trying to mitigate this issue. I really do.
Me too. Just last week they had another problem with DNS on the client side of things: Resolving with the Google Public DNS, which most droplets use by default, didn't work reliably. I hope that they post a combined post mortem for both of those incidents.
Re: DNS Outage at DigitalOcean
#27Yeah this is pretty unfortunate. We have some big investor meetings today and this unfortunately took our marketing site offline. Hopefully they resolve this soon - it's the first time we've ever experienced an issue with their service.
We really need fail-overs in place...small team problems.
Re: DNS Outage at DigitalOcean
#28Their status page at https://status.digitalocean.com is also now giving an intermittent "500 Internal Server Error" nginx error, probably from the load. That's why you should use a service like https://www.statuspage.io for your important stuff, even though creating a status page is a fun side-project for a dev team.
So what you're saying is that instead of running their own Status Page on their own infrastructure that's reachable. They should outsource it to statuspage.io and pay another company to do it?
Re: DNS Outage at DigitalOcean
#29Yeah, tried to access our site and it was down. Really was expecting more out of Digital Ocean than to fuck up such an integral part of their infrastructure. In the future we'll be transitioning away from their DNS solution because this is unacceptable.
If your site is so critical that it can't suffer any downtime then why is it not provisioned across multiple independent platforms?
Re: DNS Outage at DigitalOcean
#30Their status page at https://status.digitalocean.com is also now giving an intermittent "500 Internal Server Error" nginx error, probably from the load. That's why you should use a service like https://www.statuspage.io for your important stuff, even though creating a status page is a fun side-project for a dev team.
So what you're saying is that instead of running their own Status Page on their own infrastructure that's reachable. They should outsource it to statuspage.io and pay another company to do it?
Yes, that's pretty standard. Availability monitoring and status reporting should be external and separate from your own infrastructure, otherwise neither may be available when you need it the most.