From @DOStatus a minute ago:
"Our engineering team has identified the issue, and are working to resolve connectivity issues to our DNS servers.... http://do.co/status"
DNS Outage at DigitalOcean
41–50 of 125 posts
Re: DNS Outage at DigitalOcean
#42People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...
I've been reading all the comments on Twitter also... like "err mai gawd I'm switching to AWS because of this" and your failure to not have a secondary DNS provider, but I highly doubt you'd switch. Then another... "Today's @digitalocean DNS #outage is a reminder to not trust your entire business to one provider. Spread the love around!" If your company is e-commerce and makes money by being 99.99% available. It's yo…
Times like this makes you realize the difference betweeen good clients and bad clients. Yes, they have a right to be upset but claims like "could be losing thousands of dollars" is mostly exagerrated due to their frustration.
Re: DNS Outage at DigitalOcean
#43Earlier quoted context omitted.
It's not hard, the problem is everything relies on DNS so when DNS goes down or has problems you have cascading failure.
That's why you use multiple providers.
Suppose you have multiple providers, but one of them screws up and authoritatively denies the existence of all of your hosts?
Re: DNS Outage at DigitalOcean
#44Earlier quoted context omitted.
I hope your clients/users are as understanding and civil as you are. In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).
The problem is, for an early stage startup incidents like this are deadly. Especially since we just applied to a bunch of accelerators.
I get that this is a pain in the ass. I've got a significant chunk of infrastructure on DO, I've got work to do today that depends on those machines. I learned about this simultaneously when a deployment failed and I got a text from an engineer at a company I consult with. Not a great way to start the day, for sure.
Know what I'm going to do? I'm going to have a cup of coffee and play with my dogs for a bit. It's inconvenient, it's going to delay things, and I'm a bit choked about it. But it's not worth getting angry over, because there's nothing I can do about it today.
Re: DNS Outage at DigitalOcean
#45Feeling the pain here too. What DNS providers do others use and like? Route53?
dnsmadeeasy for around 8 years now
Re: DNS Outage at DigitalOcean
#46Earlier quoted context omitted.
So what you're saying is that instead of running their own Status Page on their own infrastructure that's reachable. They should outsource it to statuspage.io and pay another company to do it?
Yes, that's pretty standard. Availability monitoring and status reporting should be external and separate from your own infrastructure, otherwise neither may be available when you need it the most.
And don't use statuspage.io if your host is AWS, because theirs is too.
Re: DNS Outage at DigitalOcean
#47Earlier quoted context omitted.
The problem is, for an early stage startup incidents like this are deadly. Especially since we just applied to a bunch of accelerators.
The resolution is, for any app/startup/business everything is a risk and if you didn't include the edge-case of "What happens if my primary DNS nameserver goes down for my domain?" into account. Is all you can do is blame DO? If your app goes down do you have failover for that? Or do you blame your devops team?
We have auto failover for server, app, and database failures -- this can be easily managed.
DNS Nameserver failover should have been built-in, after all there is a reason we specify 3 DNS nameservers into the domain configuration, since Digital Ocean took on this task (whereby we used Gandi's DNS before that) we expected it to perform as advertised -- so yes blame lies with DO.
Re: DNS Outage at DigitalOcean
#48Earlier quoted context omitted.
I hope your clients/users are as understanding and civil as you are. In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).
The problem is, for an early stage startup incidents like this are deadly. Especially since we just applied to a bunch of accelerators.
Be happy that this happend early. Now you know that you should never ever have a single point of failure.
Re: DNS Outage at DigitalOcean
#49I would be interested in the post-mortem from this. While DigitalOcean operate their own DNS, it is only made publicly available though CloudFlares DNS proxying service.
Re: DNS Outage at DigitalOcean
#50Earlier quoted context omitted.
I've been reading all the comments on Twitter also... like "err mai gawd I'm switching to AWS because of this" and your failure to not have a secondary DNS provider, but I highly doubt you'd switch. Then another... "Today's @digitalocean DNS #outage is a reminder to not trust your entire business to one provider. Spread the love around!" If your company is e-commerce and makes money by being 99.99% available. It's yo…
Times like this makes you realize the difference betweeen good clients and bad clients. Yes, they have a right to be upset but claims like "could be losing thousands of dollars" is mostly exagerrated due to their frustration.
heh yeah, I just laugh at all the tweets saying their losing $billions of dollars every minute their site/app is unavailable. All I can think is... if you're the next Amazon.com I'm pretty sure you'd have some type of disaster plan in place should something like this happen.