A few years ago Slicehost had a DNS outage and the webscrapers I had running were falling over because they couldnt resolve DNS. I had to SSH into 8 boxes and update resolv.conf to add google DNS and openDNS as a backup. (Yes, I should've had centralized config management with chef or puppet or ansible)
DNS Outage at DigitalOcean
51–60 of 125 posts
Re: DNS Outage at DigitalOcean
#52A few years ago Slicehost had a DNS outage and the webscrapers I had running were falling over because they couldnt resolve DNS. I had to SSH into 8 boxes and update resolv.conf to add google DNS and openDNS as a backup. (Yes, I should've had centralized config management with chef or puppet or ansible)
That's crazy... I think by default DO droplets use Google DNS for resolving.
Re: DNS Outage at DigitalOcean
#53Re: DNS Outage at DigitalOcean
#54Earlier quoted context omitted.
Yes, that's pretty standard. Availability monitoring and status reporting should be external and separate from your own infrastructure, otherwise neither may be available when you need it the most.
And don't use statuspage.io if your host is AWS, because theirs is too.
Eh, as I remember it they have stuff in multiple AWS regions. Would take a global AWS failure to bring them down. This was my justification for going with them while working somewhere that used AWS.
That said, while it is extremely unlikely it shouldn't be discounted as impossible.
Re: DNS Outage at DigitalOcean
#55Earlier quoted context omitted.
Bind, running on VMs. Not hard.
You run your own authoritative DNS servers?
I ran my non-authoritative DNS server [bind] on a droplet for about a year. But the server crashed every few months. Why? Never figured out. A restart always fixed it.
Later shifted to DO's DNS servers.
Now that that one is down too, just shifted back to domain register's DNS.
Everything is working now.
Re: DNS Outage at DigitalOcean
#56Earlier quoted context omitted.
Yes, that's pretty standard. Availability monitoring and status reporting should be external and separate from your own infrastructure, otherwise neither may be available when you need it the most.
And don't use statuspage.io if your host is AWS, because theirs is too.
They do have geo-region (not just AZ) redundancy and failover, which puts them quite a bit above most home-grown company status pages in my experience. But yes, if the problem was for example in R53 it would indeed be better to have an solution without that AWS dependency.
Re: DNS Outage at DigitalOcean
#57Yeah this is pretty unfortunate. We have some big investor meetings today and this unfortunately took our marketing site offline. Hopefully they resolve this soon - it's the first time we've ever experienced an issue with their service. We really need fail-overs in place...small team problems.
We feel the pain as well as our platform is unreachable. I'm now using an other DNS server and changed the nameserver in the domain-record. However the DNS propagation is taking some time. What are you doing at the moment as fail-over?
Sorry if I'm trying to "teach grandma to suck eggs" but can't you just enter the domain in your local hosts file. If it's a network that needs access then presumably you have some sort of proxy/cache that could be seeded with the necessary domain+IP pairing? I suppose these aren't possible if you're trying to demo on someone else's network or in a public space or such.
Re: DNS Outage at DigitalOcean
#58Re: DNS Outage at DigitalOcean
#59Re: DNS Outage at DigitalOcean
#60Recommend AWS Route53 very highly. Route53 also allows you to buy domain names and do lot's of fancy fail-over, geolocation, and CNAME alias at the apex magic.