Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DNS Outage at DigitalOcean

status.digitalocean.com

Post reply on HN
Print view View original

101–110 of 125 posts

Re: DNS Outage at DigitalOcean

#101

Post by colinbartlett »

camikazeg wrote: post #87
colinbartlett wrote: post #75

If you want to get alerted when it comes back up, or you wish you had been alerted when it went down, check out my project: https://StatusGator.com . StatusGator monitors status pages and sends notifications via email, Slack, and others. You can get alerted to status changes inside Slack and you can ask it the status of a service with a /statuscheck command.

A bit of feedback: you should have a link back to your dashboard on every page. That seems like the most important page to me as a user, but if I am changing my notification or account settings, there is no way back to that page.

Great feedback, thank you! Added that.

Re: DNS Outage at DigitalOcean

#102

Post by stevekemp »

cleaver wrote: post #91
tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

I had one customer on DO DNS and it was a "good enough" solution. Unfortunately, this came right in the middle of a marketing push for last-minute registrations. An annoyance, but not a major financial impact. (Maybe it will give the impression of excess demand. :) I understand that things break and I should be ready for it. What I found unacceptable were the status updates. Basically, "we're working on it". No clue…

Route53 is a wonderful service, which has never had a global outage (so far).

I've wrapped it in git, to allow very quick updates to be made in a simple fashion:

https://dns-api.com/

Re: DNS Outage at DigitalOcean

#103

Post by stevekemp »

showerst wrote: post #8

Feeling the pain here too. What DNS providers do others use and like? Route53?

Route53 is hard not to love; simple to develop against and very very reliable.

I wrap it in git to make updates more straightforward for people unfamiliar with AWS, but even using it directly is very simple from multiple languages. (https://dns-api.com/)

Re: DNS Outage at DigitalOcean

#104

Post by mrideout »

fredophile wrote: post #100

I don't have anything more important than a small personal website but now I'm curious. If you set up a system to handle your main DNS provider failing, how do you test it? Is there a good reference where I can find some best practices on this?

Here's my testing recommendation:

1. Pick some subset of your DNS records to monitor, or all of them if you want to be extra thorough. If you are picking a subset, then I'd pick whatever records are most critical to your business.

2. Setup monitoring that queries each of your authoritative name servers for each of the records that you identified in the previous step. The monitoring should notify you if any of the name servers are unresponsive, or return a different response than what's expected.

If you'd like to dig into the details of DNS, then O'Reilly's "DNS and BIND" is highly recommended, even if you're not using BIND.

There are a number of quality hosting providers out there. A rule of thumb that I use is this: If a DNS hosting provider doesn't eat their own dog food, don't trust them to handle your DNS. Digital Ocean doesn't use their own name servers for their main website's domain. Neither does Amazon.

Shameless plug: I created a DNS monitoring service that can be used used for monitoring each of your name servers: https://www.dnscheck.co/

Re: DNS Outage at DigitalOcean

#105

Post by traviswingo »

scurvy wrote: post #99
traviswingo wrote: post #65

Earlier quoted context omitted.

We just switched it over to Route53 and set up some fail-overs there. Took us 5 mins and we're back online. Looks like DO is still offline so it seems to have been a good call...

You're back online from your perspective. What about all the name servers that have your SOA cached still looking at DO? You're still down for them.

Tough shit, lol. We now have reduced TTL times for future occurrences, but there's nothing we can do for those users who are still experiencing an outage.

Re: DNS Outage at DigitalOcean

#106

Post by IgorPartola »

tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

I use dns.he.net for my DNS hosting. It's free up to 50 zones and has been rock solid. The other day I started having some trouble with accessing some of my domain names. Turned out that all of their DNS was down and was returning NXDOMAIN for pretty much any request, including their own domains. Oops. So I emailed their support (which is usually very quick to respond and is better than I have seen with lots of paid products). Well, it then occurred to me that I will not get a response since the MX records for my domain were also hosted with them. Double oops.

On the plus side, in the past 4 years that I've used them this was the very first issue, and they fixed it within a couple of hours.

Anyone have any good recommendations on cheap or free backup DNS hosting?

Re: DNS Outage at DigitalOcean

#107

Post by nullrouted »

IgorPartola wrote: post #106
tonylemesmer wrote: post #13

People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...

I use dns.he.net for my DNS hosting. It's free up to 50 zones and has been rock solid. The other day I started having some trouble with accessing some of my domain names. Turned out that all of their DNS was down and was returning NXDOMAIN for pretty much any request, including their own domains. Oops. So I emailed their support (which is usually very quick to respond and is better than I have seen with lots of paid…

Use www.dnsmadeeasy.com and then dns.he.net as your secondary dns service or vice-versa. They will do transfers/updates from each other and work just fine.

Re: DNS Outage at DigitalOcean

#108

Post by doublerebel »

No offense to anyone here, but what is DO's SLA? Last time I looked, they did not have one.

DO is cheap for a reason. And that's the same reason I don't host with them, I can get SLA-backed infrastructure for a reasonable price and would have no excuse to my customers or cofounders.

Re: DNS Outage at DigitalOcean

#109

Post by tonyarkles »

mrideout wrote: post #104
fredophile wrote: post #100

I don't have anything more important than a small personal website but now I'm curious. If you set up a system to handle your main DNS provider failing, how do you test it? Is there a good reference where I can find some best practices on this?

Here's my testing recommendation: 1. Pick some subset of your DNS records to monitor, or all of them if you want to be extra thorough. If you are picking a subset, then I'd pick whatever records are most critical to your business. 2. Setup monitoring that queries each of your authoritative name servers for each of the records that you identified in the previous step. The monitoring should notify you if any of the nam…

The flipside to the dog food point: if DigitalOcean did use their own nameservers for their main site, then we wouldn't have been able to see the status page.

Re: DNS Outage at DigitalOcean

#110

Post by mrideout »

tonyarkles wrote: post #109
mrideout wrote: post #104

Earlier quoted context omitted.

Here's my testing recommendation: 1. Pick some subset of your DNS records to monitor, or all of them if you want to be extra thorough. If you are picking a subset, then I'd pick whatever records are most critical to your business. 2. Setup monitoring that queries each of your authoritative name servers for each of the records that you identified in the previous step. The monitoring should notify you if any of the nam…

The flipside to the dog food point: if DigitalOcean did use their own nameservers for their main site, then we wouldn't have been able to see the status page.

Good point!
Post reply on HN