In the UK here and Twitter and Github just went dark.
And playstation network. I can't play online and it's Friday :(
Massive Dyn DNS outage
271–280 of 302 posts
Re: Massive Dyn DNS outage
#272Re: Massive Dyn DNS outage
#273I have a shop on etsy. It is functional now.
Re: Massive Dyn DNS outage
#274I have a shop on etsy, and it is functional now.
Re: Massive Dyn DNS outage
#275I have a shop on etsy.com. It is functional at this time (1418 PDT).
Re: Massive Dyn DNS outage
#276Remind me to populate my /etc/hosts file after this blows over.
Re: Massive Dyn DNS outage
#277Re: Massive Dyn DNS outage
#278Earlier quoted context omitted.
loss of twitter, soundcloud, and github is what passes for tragic these days?
As a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.
I mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.
Re: Massive Dyn DNS outage
#279I thought DNS (particularly public) was basically immune to DDoS? If one DNS server is down, use the cached result or another server. DNS is some of the most distributable, cachable data I can imagine.
True, but DNS records have TTLs that expire, which forces you to reach back out to the authoritative server for the zone. A very decent number of records will have a TTL of 1hr, which means that a total DNS server outage lasting longer than an hour = 100% failure rate. You could always increase the TTL to a day (or whatever other arbitrary value), but that means that you'll be waiting upwards of a day for clients to see any updates to your DNS records. It's a tradeoff between different types of resiliency.
This is a simple case of a DNS provider evidently not having their crap together when it comes to a DDoS of any decent size. There are plenty of other DNS providers out there who are working fine currently, this DDoS is limited to just the one. It's all of that company's clients that are impacted.
I'm sure that DynETC will post something afterwards about how this was the largest DDoS they've ever encountered by many orders of magnitude and that there was no way for anyone to ever be prepared to take so much traffic... but at this point I think they're inept.
edit: just came back up. The TTLs on a few common records explain why this was so obviously a problem...
- www.reddit.com: 300 seconds (pointer to a fastly.net address, which is hosted via dynect)
- prod.reddit.map.fastlylb.net: 30 seconds
- api.twitter.com: 300 seconds
- herokussl.com: 3600 seconds
So, an outage of five minutes is enough to take out access to all of reddit and twitter. An outage of an hour is enough to take out heroku. Of course, these are best case scenarios: in reality, 50% of your users would lose access to reddit after 15 seconds (the fastly record), and so on.
Re: Massive Dyn DNS outage
#280I thought DNS (particularly public) was basically immune to DDoS? If one DNS server is down, use the cached result or another server. DNS is some of the most distributable, cachable data I can imagine.
The problem here lies with the authoritative nameservers. You have to retrieve the record from somewhere initially. The trouble here is that since these companies are using Dyn and only Dyn to be their Nameservers. When a dns request is made for github.com they're attempting to contact ns3.p20.dynect.net or another nameserver provided by Dyn. So yes caching will work for a little while, but only for the length of the TTL which your client is designed to respect. Once that TTL expires, your client and upstream DNS provider will attempt to contact that nameserver for a fresh record. Since you can't contact the nameservers, the website is effectively offline for all name resolution. Now if you know the record you can always forge the record locally in /etc/hosts or in a local dns resolver which allows for overrides. Hope this helps.