DDoS Attack Against Dyn Managed DNS • Hacker News Board

Re: DDoS Attack Against Dyn Managed DNS

#541

Post by Animats » Fri, Oct 21, 2016, 8:40 PM UTC

Animats wrote: post #458

So who was prepared for this? Pornhub: pornhub.com: Name Server: ns1.p44.dynect.net Name Server: ns2.p44.dynect.net Name Server: ns3.p44.dynect.net Name Server: ns4.p44.dynect.net Name Server: sdns3.ultradns.biz Name Server: sdns3.ultradns.com Name Server: sdns3.ultradns.net Name Server: sdns3.ultradns.org ultradns.biz: Name Server: PDNS196.ULTRADNS.ORG Name Server: ARI.ALPHA.ARIDNS.NET.AU Name Server: ARI.BETA.ARIDN…

Twilio just dumped Dyn, and is now available again.

twilio.com:

    Name Server: ns3.dnsmadeeasy.com
    Name Server: ns2.dnsmadeeasy.com
    Name Server: ns4.dnsmadeeasy.com
    Name Server: ns1.dnsmadeeasy.com
    Name Server: ns0.dnsmadeeasy.com

Re: DDoS Attack Against Dyn Managed DNS

#542

Post by detaro » Fri, Oct 21, 2016, 8:40 PM UTC

warmwaffles wrote: post #479

QUFB wrote: post #468

Earlier quoted context omitted.

Really? Route53 on AWS is $0.50/zone and $0.40/million queries. API integration is also very easy. Using something like Route53 as a backup is significantly cheaper than suffering from the current Dyn outage.

That is not helpful if you want vanity name servers

I assume your clients would prefer working nameservers over vanity ones. Especially if you are in a critical business like PagerDuty.

Re: DDoS Attack Against Dyn Managed DNS

#543

Post by lips » Fri, Oct 21, 2016, 8:41 PM UTC

How many DNS services ala Dyn exist? Is it not still massively significant that a successful attack can be launched on even one of these?

Re: DDoS Attack Against Dyn Managed DNS

#544

Post by dandandan » Fri, Oct 21, 2016, 8:44 PM UTC

warmwaffles wrote: post #487

mancerayder wrote: post #484

Earlier quoted context omitted.

Running a redundant DNS provider is expensive as all hell. While 'expensive' is a relative term, I disagree that it's cost-prohibitive for most firms, as I looked into this specifically (ironically considered using Dyn as our secondary). The challenge isn't coming up with the funds, it's if you happen to use 'intelligent DNS' features; these are proprietary (by nature) and thus they don't translate 1:1 between provid…

Ah so make everything redundant. Double my costs in man hours and in monetary cost. Brilliant!

Your automation should be handling creating/modifying records in both providers. Also, if you're utilizing multiple providers you don't need to pay for 100% of your QPS (or whatever metric is used for billing) on every provider, only 50% for two or 33% for three. You can just pay for overages when you need to send a higher percentage of your traffic to a single provider.

Re: DDoS Attack Against Dyn Managed DNS

#545

Post by pfarnsworth » Fri, Oct 21, 2016, 8:45 PM UTC

cjbprime wrote: post #474

tim_armandpour wrote: post #413

I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues. We sincerely apologize f…

The outage started more than eight hours before you posted this message..

To be fair, Hacker News isn't exactly the first place where companies post status messages. I actually applaud him for posting his message here.

Re: DDoS Attack Against Dyn Managed DNS

#546

Post by dmalvarado » Fri, Oct 21, 2016, 8:45 PM UTC

This may be dumb, but someone enlighten me:

If this kind of attacking does escalate, wouldn't it be possible to simply cut off requests from outside the United States at the points of entry? Basically, turning the US into an intranet?

Re: DDoS Attack Against Dyn Managed DNS

#547

Post by patrickg_zill » Fri, Oct 21, 2016, 8:50 PM UTC

nodesocket wrote: post #519

patrickg_zill wrote: post #481

Earlier quoted context omitted.

Sorry if this sounds dickish, but renting 3 servers @ $75 apiece from 3 different dedicated server companies in the USA, putting TinyDNS on them, and using them as backup servers, would have solved your problems hours ago. Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K/second p…

That's not how that sound be done. Just use a mix of two providers. Using your own servers and TinyDNS is silly for million/billion dollar companies. See MaxCDN for example who uses a mix of dns providers (AWS Route53 and NS1): ns-5.awsdns-00.com. ['205.251.192.5'] [TTL=172800] ns-926.awsdns-51.net. ['205.251.195.158'] [TTL=172800] ns-1762.awsdns-28.co.uk. ['205.251.198.226'] (NO GLUE) [TTL=172800] ns-1295.awsdns-33.…

I don't know how big PagerDuty is; IIRC over 200 employees, so, a decent size.

I was giving a bare-minimum example of how this or (some other backup solution) should have already been setup and ready to be switched over.

DNS is bog-simple to serve and secure (provided you don't try to do the fancier stuff and just serve DNS records): it is basically like serving static HTML in terms of difficulty.

That a company would have a backup of all important sites/IP addresses locally available and ready to deploy on some other service, or even be built by hand via some quickly rented servers, is I think quite a reasonable thing to have. I guess it would also be simple to run on GCE and Azure as well, if you don't like the idea of dedicated servers.

Re: DDoS Attack Against Dyn Managed DNS

#548

Post by ravenstine » Fri, Oct 21, 2016, 8:50 PM UTC

6d6b73 wrote: post #113

tobz wrote: post #99

Earlier quoted context omitted.

This is a DNS outage. If self-hosted, somewhere, you could still be screwed by having Dyn as your DNS provider. If dev-machine-hosted, then uh, your issue tracker is no longer an issue tracker. Your build server is not a build server. All the services besides Git are not meant to operate offline in a decentralized/distributed fashion. Library documentation, sure, that could be local. Otherwise, your assertion that al…

First of all I'm not saying that the whole infrastructure could be replicated, only critical parts, and parts that can be easily hosted locally. Second, pointing to a new machine is as simple as updating IP in your hosts file, or dns server. Third, you can use vmware or any other virtualization stack to replicate your infrastructure locally. In fact that's the best way to build things - create virtual network, use it…

Not that I'm saying that you're completely wrong, but you're oversimplifying the problem and the solution to it. Your last statement is not necessarily true; you have to balance the cost and how much of a PITA it is to set up and maintain vs how much money would be lost by disgruntled customers for a single rare outage. Granted, it depends on the kind of business you are running, but not every business is so fragile. In fact, I'd say that most(as in >50%) are not that fragile. Even better when you can simply put the blame on someone else, which everyone can do in our case right now.

Re: DDoS Attack Against Dyn Managed DNS

#549

Post by cakeface » Fri, Oct 21, 2016, 8:54 PM UTC

detaro wrote: post #341

cjbprime wrote: post #338

Earlier quoted context omitted.

If your DNS server is offline, is the last record it returned when it was online really the "wrong" one? There'd be no right one in that case.

Exactly. You can't know if it is still valid, so you might send clients to an IP that's now controlled by somebody else. Worst case, they know and set up a phishing site. DNS generally has been reliable enough that the trade-off is not worth it.

I like this idea. Grab a new elastic IP on AWS. Set up an EC2 listening on 80, maybe some other interesting ports, and see if anything juicy comes in. Or just respond with some canned SPAM or phishing attempt. Repeat.

Re: DDoS Attack Against Dyn Managed DNS

#550

Post by Animats » Fri, Oct 21, 2016, 8:56 PM UTC

BinaryIdiot wrote: post #516

Animats wrote: post #458

So who was prepared for this? Pornhub: pornhub.com: Name Server: ns1.p44.dynect.net Name Server: ns2.p44.dynect.net Name Server: ns3.p44.dynect.net Name Server: ns4.p44.dynect.net Name Server: sdns3.ultradns.biz Name Server: sdns3.ultradns.com Name Server: sdns3.ultradns.net Name Server: sdns3.ultradns.org ultradns.biz: Name Server: PDNS196.ULTRADNS.ORG Name Server: ARI.ALPHA.ARIDNS.NET.AU Name Server: ARI.BETA.ARIDN…

When your business depends on your infrastructure being up and running you try to prepare for anything. Then again Twitter is down so...

Twitter is still 100% on Dyn.

twitter.com:

    Name Server: NS1.P34.DYNECT.NET
    Name Server: NS4.P34.DYNECT.NET
    Name Server: NS2.P34.DYNECT.NET
    Name Server: NS3.P34.DYNECT.NET

They're probably using some geographically based DNS distribution scheme which they can't quickly move to other DNS servers.