Live data from Hacker News

DNS Outage at DigitalOcean

status.digitalocean.com

31–40 of 125 posts

Re: DNS Outage at DigitalOcean

#31
post #2

Yeah, tried to access our site and it was down. Really was expecting more out of Digital Ocean than to fuck up such an integral part of their infrastructure. In the future we'll be transitioning away from their DNS solution because this is unacceptable.

I also hope the users of your site understand that shit happens. Also as another user said... if DNS is so critical for you then why don't you have proper failover in place?

Re: DNS Outage at DigitalOcean

#34
post #2

Yeah, tried to access our site and it was down. Really was expecting more out of Digital Ocean than to fuck up such an integral part of their infrastructure. In the future we'll be transitioning away from their DNS solution because this is unacceptable.

I hope your clients/users are as understanding and civil as you are. In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).

The problem is, for an early stage startup incidents like this are deadly. Especially since we just applied to a bunch of accelerators.

Re: DNS Outage at DigitalOcean

#35

DigitalOcean uses CloudFlare for DNS - https://www.cloudflare.com/case-studies-digital-ocean/

This statement can be misleading. If you read the article, they don't use CloudFlare's DNS servers per se. They use CloudFlare's DNS proxy which acts as a DNS firewall between the DigitalOcean DNS servers and the world.

Re: DNS Outage at DigitalOcean

#36
post #10
post #6

DNS is hard. Very hard. It may seems trivial when it works (hint: it's not), but some of the biggest fuck ups I've seen in my professional life were caused by strange DNS things happening or DNS servers going kaboom. I feel the pain of the DO engineers trying to mitigate this issue. I really do.

It's not hard, the problem is everything relies on DNS so when DNS goes down or has problems you have cascading failure.

That's why you use multiple providers.

Re: DNS Outage at DigitalOcean

#37
post #2

Yeah, tried to access our site and it was down. Really was expecting more out of Digital Ocean than to fuck up such an integral part of their infrastructure. In the future we'll be transitioning away from their DNS solution because this is unacceptable.

I hope your clients/users are as understanding and civil as you are. In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).

Just add a second dns provider.

Re: DNS Outage at DigitalOcean

#39

Earlier quoted context omitted.

I hope your clients/users are as understanding and civil as you are. In the meantime, I'm going wait for post-mortem before deciding if I should continue using them for dns. Looking back over the status history, 1-2 incidents a year isn't that bad for my needs, but might be too much for you, which is fine (since I'm only hosting a couple of small side projects with them).

The problem is, for an early stage startup incidents like this are deadly. Especially since we just applied to a bunch of accelerators.

The resolution is, for any app/startup/business everything is a risk and if you didn't include the edge-case of "What happens if my primary DNS nameserver goes down for my domain?" into account. Is all you can do is blame DO?

If your app goes down do you have failover for that? Or do you blame your devops team?

Re: DNS Outage at DigitalOcean

#40

Yeah this is pretty unfortunate. We have some big investor meetings today and this unfortunately took our marketing site offline. Hopefully they resolve this soon - it's the first time we've ever experienced an issue with their service. We really need fail-overs in place...small team problems.

We feel the pain as well as our platform is unreachable. I'm now using an other DNS server and changed the nameserver in the domain-record. However the DNS propagation is taking some time. What are you doing at the moment as fail-over?
Post reply on HN