This is the first DNS outage I've experienced with them in 3+ years, then again I host everything in their NY regions.
Same. They've been pretty reliable. Hoping this doesn't last long or we'll be looking to move off.
DNS Outage at DigitalOcean
11–20 of 125 posts
Re: DNS Outage at DigitalOcean
#12DigitalOcean uses CloudFlare for DNS - https://www.cloudflare.com/case-studies-digital-ocean/
Re: DNS Outage at DigitalOcean
#13People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable.
[1]https://twitter.com/rodrigoespinosa/status/71303563702097100...
Re: DNS Outage at DigitalOcean
#14Feeling the pain here too. What DNS providers do others use and like? Route53?
The sites I have that are actually up right now are those routed through CloudFlare.
Re: DNS Outage at DigitalOcean
#15My rule: provider should do single thing:
- Hosting provider - host sites
- vps/cloud provider - provide VMs
- domain registrar - domain related stuff, but not DNS
- dns provider - host dns
- second dns provider - host dns in case first dns provider fails
So many DNS outages recently and all my projects are up.
Re: DNS Outage at DigitalOcean
#16Does anyone know of a good strategy for DNS failover?
I don't know if DigitalOcean's DNS servers allow AXFR, if they do you can use a secondary DNS service to automatically replicate the DNS. You then list the secondary DNS as a NS for your domain.
If they don't allow AXFR -- and after this, they should! -- you can still have a secondary DNS provider but you'd have to duplicate any changes by hand. Not ideal but still doable.
Re: DNS Outage at DigitalOcean
#17Does anyone know of a good strategy for DNS failover?
[deleted]
Re: DNS Outage at DigitalOcean
#18Does anyone know of a good strategy for DNS failover?
Well, there's a couple of strategies:
- IP-diverse nameservers
- TLD-diverse nameservers
- BGP anycast
IP-diverse nameservers requires that you expect that your DNS servers will go down rather than start returning bad results - I highly recommend having some sort of mechanism to hard-terminate access to those machines.
TLD-diverse nameservers is just an extra strategy for reducing the risk that an upstream TLD issue will blow up your spot.
And then BGP anycast is the expensive, complicated piece of this - it requires a high level of technical sophistication, lots of moving parts, and the QA/validation piece of it is tricky.
When I built an anycast DNS system, we ended up resorting to tricks like having the DNS servers publish routes to the router for redistribution, so that a down or unresponsive server automatically withdrew the routes. Then you do things like TXT records for your zone that respond with which POP you're hitting in some sort of hashed/obfuscated fashion.
It's hard and complicated, and unnecessary for most folks. Better to outsource to Route 53 or someone similar.
Re: DNS Outage at DigitalOcean
#19Feeling the pain here too. What DNS providers do others use and like? Route53?
Where I work we use Rotue53. For my personal domains I just use my registrar, Namecheap.
For us, Route53 is painful. We host a few thousand zones, and due to rate limiting on APIs, doing something like "Show a list of domain names" or "Give all the domains matching some pattern" were particularly painful. Upwards of 30 seconds to do a simple list of domains meant we were forced to cache locally. A local cache, combined with the fact that zone names are not unique in their system (possible to create multiple abc.com entries, which differ only in an internal id and the list of NS entries) made it hard to ensure that our internal systems matched "reality". Then the administrative nightmare of 3-4 different NS entries for each zone means customized, rather than generic, instructions for validating NS settings at the individual registrars.
All in all, it was not a fun experience with such a large volume of zones, but we knew we were an edge case.
Re: DNS Outage at DigitalOcean
#20People hating on DO "I'm losing thousands every hour". Well then should have had some failover in place if its that valuable. [1] https://twitter.com/rodrigoespinosa/status/71303563702097100...
I've been reading all the comments on Twitter also... like "err mai gawd I'm switching to AWS because of this" and your failure to not have a secondary DNS provider, but I highly doubt you'd switch.
Then another... "Today's @digitalocean DNS #outage is a reminder to not trust your entire business to one provider. Spread the love around!"
If your company is e-commerce and makes money by being 99.99% available. It's your own fault for no fail-over.
another... ".@digitalocean that's two hours without DNS now...my company's websites could be losing thousands of £ in e-commerce! Please, an update!"