Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Stack Overflow plans to survive the next DNS attack

blog.serverfault.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: How Stack Overflow plans to survive the next DNS attack

#31

Post by thefarseeker »

matt4077 wrote: post #12

The calculation regarding the ideal number of name servers to list needs some empirical data regarding the likelihood of provider and server outages and the client reactions to it, right? Because otherwise 2 would must be the best number, if I'm not mistaken (Chance of hitting the provider that's offline is always 0.5 on the first try, but the second try would be guaranteed to hit the other). Here's the math for expe…

Author here. One thing I didn't cover in the post was about how to have Google and AWS honour their SLA's is that you have to use _all 4_ of the nameservers they provide. Because we're not doing that (only using half), we have to balance the chance of an outage versus the impact of an outage.

You are correct in saying that more empirical data could be used here. We might even end up changing our minds. I'm not much of a numbers person but I might pass this onto some of the people in our company who love solving problems like this.

Re: How Stack Overflow plans to survive the next DNS attack

#32

Post by thefarseeker »

vaara wrote: post #21

I wonder why there's no consideration of anycast servers.

Author here. Google and AWS are anycasted services. We were not interested in running our own anycasted DNS due to the management overhead of doing so, and because the cost of anycasting our own services would be orders of magnitude more expensive than outsourcing that to an established DNS provider.

Re: How Stack Overflow plans to survive the next DNS attack

#33

Post by thefarseeker »

jlgaddis wrote: post #19

It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers. Hurricane Electric supports this but most of the providers mentioned in this article do not.

Author here. I agree. There are built-in mechanisms for doing this - AXFR and IXFR. However, these mechanisms were not really designed with this sort of scale in mind. You have to keep an up to date whitelist of all the servers that can talk to each other, and they would need to talk to each other on a non-anycasted address (otherwise the notify packet would go to just a single anycasted node).

Managing whitelists between multiple 3rd party DNS providers is likely to break frequently as servers move around, are added, removed, etc.

Interestingly, Hurricane Electric would have been one of our top choices if they had a first class API and a commercial SLA. Their ability to support zone transfers is admirable and did not go un-noticed. DNS Made Easy also supports zone transfers.

Re: How Stack Overflow plans to survive the next DNS attack

#34

Post by thefarseeker »

matt4077 wrote: post #11

I think I measured Cloudflare's performance and chose it over Google because it was consistently faster. If the stack-stackers are reading, I'd love to hear why they didn't make the list. Also, it'd be a great public service to publish the results. Even if it's just enabled for a day per year or so the results would probably appreciated by many. And you could always sell your altruism as the need to continually monit…

Author here. CloudFlare's DNS was measurably, and consistently, faster than almost every provider we tested. So I can concur with you in that regards. It was ultimately faster than Route 53 and Google Cloud (the path we ended up going).

However, when our relationship with CloudFlare ended and we moved to Fastly, one of the reasons we did so was unreliable lag with DNS updates at CloudFlare. Sometimes it would take minutes or hours for DNS changes to be affected in their system, and they could never provide a satisfactory reason why.

Re: How Stack Overflow plans to survive the next DNS attack

#35

Post by ksec »

From my experience EdgeCast and DNSMadeEasy were consistently the fastest DNS. I guess both were dropped because of price when Google DNS and Route53 did the a similar job.

And as other have said, while Cloudflare may not be for everyone, their DNS is possibly the fastest. Not sure why SO decide to drop them.

*Some old Data http://www.dnsperf.com/

I also wonder on the performance of DNSimple. But they dont see to emphasis much on performance.

Re: How Stack Overflow plans to survive the next DNS attack

#36

Post by stevekemp »

jlgaddis wrote: post #19

It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers. Hurricane Electric supports this but most of the providers mentioned in this article do not.

Sadly most of the bigger providers require you to code to their API - which makes migrations a little more complex - and that goes double if you get locked into using special DNS-records (rather than common types such as A/AAAA/MX/etc).

That's one of the reasons why the DNS hosting I support, which uses git-hooks to trigger updates, only currently pushes the DNS data to Amazon's route53 infrastructure.

At the time of the most recent Dyn outage I looked at allowing users to support multiple back-ends, to abstract away the pain of redundancy, but it seemed there was surprisingly little interest.

Re: How Stack Overflow plans to survive the next DNS attack

#37

Post by thefarseeker »

ksec wrote: post #35

From my experience EdgeCast and DNSMadeEasy were consistently the fastest DNS. I guess both were dropped because of price when Google DNS and Route53 did the a similar job. And as other have said, while Cloudflare may not be for everyone, their DNS is possibly the fastest. Not sure why SO decide to drop them. *Some old Data http://www.dnsperf.com/ I also wonder on the performance of DNSimple. But they dont see to emp…

So DNSMadeEasy made it all the way through the barrage of tests. I even wrote a library for their API so that we could integrate it into our DNS software (https://github.com/mhenderson-so/godnsmadeeasy), but at the end of the day their performance in certain regions was not good enough. In some countries they were measurably faster than R53, but in others they were measurably much slower.

EdgeCast were dropped due to pricing, and that there's talk of Verizon selling the EdgeCast services again.

DNSimple didn't make it to performance testing because they only had 5 POPs, as opposed to 20+ of other providers.

CloudFlare's DNS was consistently one of the fastest, you are correct about that. If you read my responses to other comments here, you'll find that we decided not use their DNS service because of some fairly pervasive API issues we had with it.

Re: How Stack Overflow plans to survive the next DNS attack

#38

Post by jlgaddis »

thefarseeker wrote: post #33
jlgaddis wrote: post #19

It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers. Hurricane Electric supports this but most of the providers mentioned in this article do not.

Author here. I agree. There are built-in mechanisms for doing this - AXFR and IXFR. However, these mechanisms were not really designed with this sort of scale in mind. You have to keep an up to date whitelist of all the servers that can talk to each other, and they would need to talk to each other on a non-anycasted address (otherwise the notify packet would go to just a single anycasted node). Managing whitelists be…

Just as an additional data point for anyone else reading this...

Hurricane Electric supports zone transfers and requires you to only allow AXFR's from a single host -- slave.dns.he.net (IPv4: 216.218.133.2, IPv6: [2001:470:600::2]). NOTIFYs should not be sent to slave.dns.he.net but instead to ns1.he.net.

n.b.: ns1.he.net is not anycasted, but ns[2-5] are. In addition, ns1 does not have an AAAA RR.

We (ISP) currently run our own authoritative name servers in our own facilities but I've been seriously debating adding another provider into the mix so "secondary" service is an important feature to me.

Re: How Stack Overflow plans to survive the next DNS attack

#39

Post by jlgaddis »

stevekemp wrote: post #36
jlgaddis wrote: post #19

It'd be great if more DNS providers supported "slaving" a zone from an existing server. It would make it much easier to keep DNS synchronized across multiple providers. Hurricane Electric supports this but most of the providers mentioned in this article do not.

Sadly most of the bigger providers require you to code to their API - which makes migrations a little more complex - and that goes double if you get locked into using special DNS-records (rather than common types such as A/AAAA/MX/etc). That's one of the reasons why the DNS hosting I support, which uses git-hooks to trigger updates, only currently pushes the DNS data to Amazon's route53 infrastructure. At the time of…

I'm glad more and more providers are offering APIs these days but the important feature for me is the ability to slave off of my own servers.

We (ISP) run our own authoritative name servers. Ideally, I'd have a single hidden ("stealth") master (maybe two, w/ anycast) and all of the public name servers would simply slave from that one. If you run PowerDNS -- which supports MySQL/PostgreSQL backends, among others -- you can keep everything in a local database and use standard tools (or write your own) to manage it.

(If I was pretty much anywhere besides an ISP, I'd definitely be using a provider with a fully-featured API. I use Route 53 now for my personal domains but I manage the zones by hand in the console since the RRs practically never change.)

Re: How Stack Overflow plans to survive the next DNS attack

#40

Post by stevekemp »

jlgaddis wrote: post #39
stevekemp wrote: post #36

Earlier quoted context omitted.

Sadly most of the bigger providers require you to code to their API - which makes migrations a little more complex - and that goes double if you get locked into using special DNS-records (rather than common types such as A/AAAA/MX/etc). That's one of the reasons why the DNS hosting I support, which uses git-hooks to trigger updates, only currently pushes the DNS data to Amazon's route53 infrastructure. At the time of…

I'm glad more and more providers are offering APIs these days but the important feature for me is the ability to slave off of my own servers. We (ISP) run our own authoritative name servers. Ideally, I'd have a single hidden ("stealth") master (maybe two, w/ anycast) and all of the public name servers would simply slave from that one. If you run PowerDNS -- which supports MySQL/PostgreSQL backends, among others -- yo…

It would definitely be nice, I agree.

Though with a decent API it wouldn't be hard to write the glue to do it - I've certainly converted from bind to my own representation, then from that to Route53.

It's just a shame we all have to keep reinventing the wheel.

Post reply on HN