Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zerigo DNS services down for 6+ hours due to massive DDoS

zerigostatus.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#51

Post by wlll »

aeden wrote: post #30
PonyGumbo wrote: post #28

Earlier quoted context omitted.

Your pricing looks amazing - but it looks like you had an outage this morning too. Can you talk about that?

Yes, we had a short outage this morning. We're still investigating the root cause but the symptoms were essentially a simultaneous slow down across all name servers. It's quite possible this is part of the same DDoS that Zerigo is facing, we're not certain - we've been seeing lots of spikes today from various IP addresses (but we've actually been seeing similar patterns for quite some time, as have other DNS provider…

Two points:

- The downtime was 40 minutes (according to Pingdom), that isn't short for something as critical as DNS.

- It seems off to pimp yourself on a thread about a competitors downtime, especially when you had a significant one at pretty much the same time.

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#52

Post by quellhorst »

aeden wrote: post #14

I run DNSimple ( https://dnsimple.com ) and we have a full REST API and support domain registrations, transfers and SSL certificates as well. Plus we have an ALIAS record type that's very useful for pointing your apex to services where they only provide a hostname. I'll be happy to answer any questions you have regarding our service either here or through our support channels.

Maybe you should check if your own services are up next time you spam them in a thread where your competitors are down.

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#53

Post by metalruler »

I've been seeing a lot of reflector attacks in the past couple of weeks, where the attacker sends a relatively small query for a valid domain that will return a large reply. The trick is that they spoof the source IP, so the DNS reply goes to the victim.

I ended up hacking something together to firewall any IPs which sent more than 1000 requests in a short period of time.

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#54

Post by dedene »

metalruler wrote: post #53

I've been seeing a lot of reflector attacks in the past couple of weeks, where the attacker sends a relatively small query for a valid domain that will return a large reply. The trick is that they spoof the source IP, so the DNS reply goes to the victim. I ended up hacking something together to firewall any IPs which sent more than 1000 requests in a short period of time.

Do you mind sharing the script / code to accomplish that? (some gist somewhere) I'm seeing a lot of these sort of things on our servers too..

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#55

Post by metalruler »

dedene wrote: post #54
metalruler wrote: post #53

I've been seeing a lot of reflector attacks in the past couple of weeks, where the attacker sends a relatively small query for a valid domain that will return a large reply. The trick is that they spoof the source IP, so the DNS reply goes to the victim. I ended up hacking something together to firewall any IPs which sent more than 1000 requests in a short period of time.

Do you mind sharing the script / code to accomplish that? (some gist somewhere) I'm seeing a lot of these sort of things on our servers too..

It really is a disgusting hack, and specific to FreeBSD. It does need to be a bit more sophisticated than "block an IP if it floods me" because as it is now someone can simply spoof the IP of an ISP's DNS server and effectively firewall them, blocking their users from being able to resolve the domain names I'm hosting.

I can give you one tip to get you started: if you're running named, you can enable logging of every query, something like (hope this formats ok) :

  logging {
    channel query_logging {
         file "/var/log/named/querylog"
         versions 3 size 100M;
         print-time yes;                 // timestamp log entries
      };

      category queries {
          query_logging;
      };
};

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#56

Post by aeden »

wlll wrote: post #51
aeden wrote: post #30

Earlier quoted context omitted.

Yes, we had a short outage this morning. We're still investigating the root cause but the symptoms were essentially a simultaneous slow down across all name servers. It's quite possible this is part of the same DDoS that Zerigo is facing, we're not certain - we've been seeing lots of spikes today from various IP addresses (but we've actually been seeing similar patterns for quite some time, as have other DNS provider…

Two points: - The downtime was 40 minutes (according to Pingdom), that isn't short for something as critical as DNS. - It seems off to pimp yourself on a thread about a competitors downtime, especially when you had a significant one at pretty much the same time.

http://stats.pingdom.com/qqps0x9eb0at/195711 indicates the outage on NS1 was 26 minutes. It's still a long time and we're pulling the trigger on some changes (read: capital investments) to stop it from happening in the future.

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#57

Post by Rulero »

aeden wrote: post #14

I run DNSimple ( https://dnsimple.com ) and we have a full REST API and support domain registrations, transfers and SSL certificates as well. Plus we have an ALIAS record type that's very useful for pointing your apex to services where they only provide a hostname. I'll be happy to answer any questions you have regarding our service either here or through our support channels.

I signed up for your service earlier this morning because Zerigo were taking their time to solve the issue.

However, what I would like to know is - have you guys implemented any procedures to mitigate any negative effects a DDoS may have on your services? (Assuming your service gets DDoS'd like Zerigo) The last thing I want is more down time and to switch to another provider once again.

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#58

Post by genek »

wlll wrote: post #51
aeden wrote: post #30

Earlier quoted context omitted.

Yes, we had a short outage this morning. We're still investigating the root cause but the symptoms were essentially a simultaneous slow down across all name servers. It's quite possible this is part of the same DDoS that Zerigo is facing, we're not certain - we've been seeing lots of spikes today from various IP addresses (but we've actually been seeing similar patterns for quite some time, as have other DNS provider…

Two points: - The downtime was 40 minutes (according to Pingdom), that isn't short for something as critical as DNS. - It seems off to pimp yourself on a thread about a competitors downtime, especially when you had a significant one at pretty much the same time.

Instead of adding another unicast network to the mix, why wouldn't you start using an IP anycast network? DNSMadeEasy / Route53 / EasyDNS / etc.. It seems crazy to move from one service that purchases 6 name servers to another one that is only on 6 servers.... DNS Made Easy fought an attack that was over 200 Gbps a few months ago. http://www.facebook.com/photo.php?fbid=10150668694804467&#38...

Can any unicast provider even really get close to fighting an attack like this?

Let's be serious at some point.... 6 servers... the MOST you can push is 6 Gbps. And most likely they are bound to about 400 Mbps of DNS traffic (based on CPU load). Unless you have hundreds of name servers and multiple locations... are you even really competing in uptime anymore?

Re: Zerigo DNS services down for 6+ hours due to massive DDoS

#59

Post by genek »

aeden wrote: post #27
latch wrote: post #26

Earlier quoted context omitted.

I know there's much love for DNSimple, but this is the first time that I can remember when the top comment of an X is down post is a competitor essentially posting an advert with no insight on the OP.

I tried to add some insight on another comment, but it's tough to say anything about DDoS that hasn't already been said. DDoS attacks suck, mitigating them requires a multi-prong approach and proactive monitoring and agressive banning and even then you can still be screwed if your bandwidth is saturated. I feel for the operational folks at Zerigo - dealing with this type of outage is hard. The best thing they can do…

How much capacity does DNSimple have though. It appears as though you are another unicast network. ns1.dnsimple.com is a server at Slicehost / Rackspace ns2.dnsimple.com is a server at Linode ns3.dnsimple.com is a server at prgmr.com / EGIHosting / Hurricane Electric ns4.dnsimple.com is a EC2 instance on Amazon

How much computing power and attack traffic can those really handle?

If you are going to offer a solution to a massive DDoS I would think that you would be careful on when to propose your solution.

Instead of adding another unicast network to the mix, why wouldn't you start using an IP anycast network?

Please explain how much capacity you have.

Post reply on HN