February 28th DDoS Incident Report

- Volumetric attacks like this one, mostly reflection.

- Application level attacks like SYN floods or protocol-specific attacks.

Defending against both costs a LOT of money.

Volumetric attack are dealt with at the network edge using rate limits and router ACLs. They're really easy to identify and block, but the point is that you need more bandwidth than the attacker in order to successfully do so. With attacks in the terabits-per-second range, this gets expensive.

Application-level attacks are harder to execute since there's no amplification and you need more bandwidth to pull it off, but they're much harder to block, too. They exhaust the server software's capacity by mimicking a real client. Common examples are SYN or HTTP floods.

When you get hit by a DDoS attack, you have two choices:

- Filter the attack and block the offending traffic without affecting legitimate requests. This is hard, and most companies can't do this. They need to have someone like Akamai on the retainer and dynamically reroute traffic like GitHub did.

- Declare bankruptcy and announce a blackhole route to your upstream providers (taking down the host in question, but protecting the rest of your network).

When you host custom applications that can't be scaled out or cached, DDoS mitigation is especially hard since you cannot just throw more servers at it like CloudFlare does.

Most services we host use proprietary binary UDP protocols, which is unfortunate, since UDP is easy to spoof and even experienced DDoS mitigation companies have trouble filtering it. Our customers get hit by DDoS attacks 24/7, so blackholing is not an option.

We had to build our own line-rate filtering appliances in order to handle the ever-increasing number of application-level DDoS attacks, by reverse engineering the binary protocols and building custom filtering and flow tracking.

All of this costs a huge amount of money, and most ISPs simply lack the resources to do this.

Happy to answer questions, but I'm going home right now, so it may take a few hours :-)

(Nitrado is a leading hosting provider specializing on online gaming, both for businesses/studios and regular customers, so we're dealing with DDoS attacks on a regular basis. We got hit with the same memcached attacks than GitHub and CloudFlare, and it was the largest attack in our company history. Ping me if you want to talk.)