I wonder why there's no consideration of anycast servers.
How Stack Overflow plans to survive the next DNS attack
21–30 of 41 posts
Re: How Stack Overflow plans to survive the next DNS attack
#22I think I measured Cloudflare's performance and chose it over Google because it was consistently faster. If the stack-stackers are reading, I'd love to hear why they didn't make the list. Also, it'd be a great public service to publish the results. Even if it's just enabled for a day per year or so the results would probably appreciated by many. And you could always sell your altruism as the need to continually monit…
+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN. Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare? [1] http://nickcr…
The fact that they abandoned Cloudflare only 6 months after this post means that they must've been pretty disappointed. I wonder if this is the same reason as for some of the other pages that went to Fastly from Cloudflare (e.g. Imgur).
Re: How Stack Overflow plans to survive the next DNS attack
#23Earlier quoted context omitted.
+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN. Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare? [1] http://nickcr…
tl;dr: Javascript/Google reCaptcha Paywall. Source: https://meta.stackoverflow.com/questions/323537/cloudflare-i...
They have a DNS service which can be used independently of their caching.
Re: How Stack Overflow plans to survive the next DNS attack
#24It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers. Hurricane Electric supports this but most of the providers mentioned in this article do not.
This'd be awesome, and there are very few providers which offer this. When looking into this, I could only find HE and BuddyNS.
Everyone seems to be inventing their own custom API for this, which I guess is the 'modern developer friendly' approach, but it results in a bit of a mess. Example: Caddy's implementation of the Let's Encrypt / ACME dns-01 challenge has all these plugins: https://caddyserver.com/download
We ended up running our own authoritative nameservers, which is not ideal. But at least cloud offerings allow you to spread across regions.
Re: How Stack Overflow plans to survive the next DNS attack
#25I think I measured Cloudflare's performance and chose it over Google because it was consistently faster. If the stack-stackers are reading, I'd love to hear why they didn't make the list. Also, it'd be a great public service to publish the results. Even if it's just enabled for a day per year or so the results would probably appreciated by many. And you could always sell your altruism as the need to continually monit…
+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN. Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare? [1] http://nickcr…
Can't believe they're running on Windows. Yikes.
Re: How Stack Overflow plans to survive the next DNS attack
#26Earlier quoted context omitted.
tl;dr: Javascript/Google reCaptcha Paywall. Source: https://meta.stackoverflow.com/questions/323537/cloudflare-i...
They have a DNS service which can be used independently of their caching.
I came to say the same thing. I use Cloudflare as a DNS provider for my own site as I was impressed by the numbers. It's literally as straightforward as clicking a button to prevent Cloudflare from proxying traffic to use their service only for nameservers.
Pretty sure the captcha can be switched off for enterprise clients (which I presume SO would be).
Regarding caching, SO's caching is extremely aggressive. It's especially problematic when editing answers more than once, because when you click edit you'll be presented with a cached copy of your answer, potentially excluding your latest revisions. So you have to refresh the edit page and then edit the answer.
It might seem like an edge case, and it certainly teaches you to be meticulous about answering lest you have to go through the refresh hell, but it's not what some would designate as 'good UX'.
I am never surprised by SO's reports about how they run their busy website from a mere handful of machines precisely because of the caching they do.
Re: How Stack Overflow plans to survive the next DNS attack
#27Earlier quoted context omitted.
+1 - I am also very interested why they chose to switch out from CloudFlare for both their DNS and their CDN and over to Fastly. Nick Craver did a write up where they specifically mentioned [1] Cloudflare for both their DNS and CDN. Do you think after the Dyn outage everyones sysadmins are running round adding redundancy, too worried to trust the uptime of their site in the hands of just CloudFlare? [1] http://nickcr…
The fact that they abandoned Cloudflare only 6 months after this post means that they must've been pretty disappointed. I wonder if this is the same reason as for some of the other pages that went to Fastly from Cloudflare (e.g. Imgur).
It could also be due to some mundane detail like the service's cost.
Re: How Stack Overflow plans to survive the next DNS attack
#28Earlier quoted context omitted.
The fact that they abandoned Cloudflare only 6 months after this post means that they must've been pretty disappointed. I wonder if this is the same reason as for some of the other pages that went to Fastly from Cloudflare (e.g. Imgur).
It could also be due to some mundane detail like the service's cost.
He mentioned 503 errors and "missed deadlines". I don't think it's costs, they were known beforehand, you wouldn't switch after 6 months because of costs. And I think if you're at that scale, you'd get a counteroffer from Cloudflare if you threaten to leave.
As you can whitelist Tor traffic in Cloudflare, it seems to be down to these 503 errors (edge to origin). But haven't heard that before, so not sure if it's a problem that occurs more often.
For Imgur, I could imagine that purging by cache tag is just too restrictive at Cloudflare (the limit is very low, even for Enterprise clients). Fastly doesn't have a limit there, they encourage you to cache everything and purge where needed. Makes it much easier to cache APIs and HTML pages.
Re: How Stack Overflow plans to survive the next DNS attack
#29I think I measured Cloudflare's performance and chose it over Google because it was consistently faster. If the stack-stackers are reading, I'd love to hear why they didn't make the list. Also, it'd be a great public service to publish the results. Even if it's just enabled for a day per year or so the results would probably appreciated by many. And you could always sell your altruism as the need to continually monit…
One major issue with their dns was unpredictable lag while changing records. On most occasions we can change a record and it would propigate immediately. In some rare cases, it could take fifteen minutes, and in a few instances up to eight hours before the change would show up on their nameservers.
Cloudflare was unable to diagnose this or give us any comfort that it would improve.
Re: How Stack Overflow plans to survive the next DNS attack
#30Umm, brilliant thank you for this. I ended up with a Dyn / Route53 configuration. We used libcloud to sync everything together. We also added the exported zone to Cloudflare but did not enable it. We had actually planned for this, but in no way did we ever come close to your in depth testing. The @ Azure issue - thank you for uncovering this for the rest of us.
Can you maybe share how you did the sync between them? Also there are already some tools pulling zone data from Dyn and adding it to Route53? Can you share why did you choose an own sync. (I'm planning to do the same and I'm interested in other's opinions)
We actually wrote a tool to manage this. We define our desired records in a common dsl format, and the tool can interact with various providers to ensure things match the expected state.
We should be open sourcing this rather shortly, so stay tuned.
Sorry, I'm not who you asked, but that is how we are doing it at stack overflow now.