Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DNS Outage at DigitalOcean

status.digitalocean.com

Post reply on HN
Print view View original

91–100 of 125 posts

Re: DNS Outage at DigitalOcean

#91

Post by cleaver »

tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

I had one customer on DO DNS and it was a "good enough" solution. Unfortunately, this came right in the middle of a marketing push for last-minute registrations. An annoyance, but not a major financial impact. (Maybe it will give the impression of excess demand. :)

I understand that things break and I should be ready for it. What I found unacceptable were the status updates. Basically, "we're working on it". No clue as to what was going on. A DDoS? Not a DDoS? Routing issues? Corrupt zone files? No clue? Any of those would be helpful as I needed to figure out if I should wait it out, or switch to Route 53.

In the information vacuum, I switched to Route 53. It works.

Re: DNS Outage at DigitalOcean

#93

Post by tracker1 »

colinbartlett wrote: post #79
tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

I can't disagree with what you're saying, but I think we are all guilty of this. We expect more out of big name services than might be reasonable. (100% uptime) How many of us here have failover email services in case Gmail goes down? I think many companies would say they'd lose thousands in productivity if Google Apps suffers an outage yet I'd hazard that very few have failover plans.

That's because people like to complain... the reality is stuff happens, systems go down, and life tends to go on.

Yeah, if your building full of employees can't work because the internet is down, and the secondary is also down, then that's kind of crappy, and you may be paying people to twiddle their thumbs... much short of that, it's kind of the cost of doing business...

There are redundancy options for a lot of things... If you're using only a single host provider for your infrastructure, and management scoffs at creating redundant, and under-utilized systems... it's not as "mission critical" as people think/say.

Re: DNS Outage at DigitalOcean

#94

Post by troydavis »

coreyp_1 wrote: post #7

Does anyone know of a good strategy for DNS failover?

A fair number of providers support zone transfers (AXFR requests) from master to slave name servers. The slaves can be operated by a different entity.

Here's DNSimple's implementation: https://support.dnsimple.com/articles/secondary-dns/

I wrote about moving from 1 to 2+ authoritative DNS providers: http://blog.papertrailapp.com/dns-outage-on-monday-december-.... I think this is just as true today:

> For .. maintainers of mission-critical DNS zones, the solution is to not depend on any single DNS infrastructure for functioning authoritative DNS

Re: DNS Outage at DigitalOcean

#95

Post by tracker1 »

Maakuth wrote: post #74
brndn wrote: post #70

Earlier quoted context omitted.

If you are showing a demo or something, you can still navigate to your DO IP address. Of course, I don't know if other things (images, etc.) on your website also rely on their DNS.

For that kind of emergency, you can point the domain name to the IP address via /etc/hosts (there's a similar file in Windows as well).

c:\windows\system32\drivers\etc\hosts

Re: DNS Outage at DigitalOcean

#96

Post by scurvy »

bitJericho wrote: post #36
Thaxll wrote: post #10

Earlier quoted context omitted.

It's not hard, the problem is everything relies on DNS so when DNS goes down or has problems you have cascading failure.

That's why you use multiple providers.

I'll bite. You can have multiple NS records but only a single SOA. The .com registry minimum TTL for SOA is a day.

How in the world would "multiple providers" help you in a 6 hour outage?

Re: DNS Outage at DigitalOcean

#97

Post by scurvy »

tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

How do you fail over your SOA on .com when the minimum TTL is 1 day?

Re: DNS Outage at DigitalOcean

#98

Post by jessaustin »

colinbartlett wrote: post #73
clentaminator wrote: post #62

Earlier quoted context omitted.

But who monitors the status of statuspage.io?

I know you're joking, but I do: Check out https://StatusGator.com . StatusPage.io has a status page at metastatuspage.com which my company monitors.

Are you also on AWS? Apparently you're not on DO...

Re: DNS Outage at DigitalOcean

#99

Post by scurvy »

traviswingo wrote: post #65
defenestration wrote: post #40

Earlier quoted context omitted.

We feel the pain as well as our platform is unreachable. I'm now using an other DNS server and changed the nameserver in the domain-record. However the DNS propagation is taking some time. What are you doing at the moment as fail-over?

We just switched it over to Route53 and set up some fail-overs there. Took us 5 mins and we're back online. Looks like DO is still offline so it seems to have been a good call...

You're back online from your perspective. What about all the name servers that have your SOA cached still looking at DO? You're still down for them.

Re: DNS Outage at DigitalOcean

#100

Post by fredophile »

I don't have anything more important than a small personal website but now I'm curious. If you set up a system to handle your main DNS provider failing, how do you test it? Is there a good reference where I can find some best practices on this?
Post reply on HN