Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

61–70 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#61

This may be a dumb question as I haven't read the bill, but isn't this what CISA is supposed to help guard against in the future? If Linode has an easier way of sharing info with the USG, can that help pinpoint and mitigate this attack from happening in the future? Edit: how about a response instead of downvotes?

DDoSes are easy because ISPs don't want to filter traffic (BCP 38). If ISPs did this, then it wouldn't be so easy to spoof source IPs and much easier to block or track down offenders. I don't see how CISA is gonna change this. If the USG wanted to stop this stuff, they'd create a policy forcing ISPs to filter. Same as VoIP and fake caller ID. If the FCC really cared, they'd start fining.

Re: Linode is suffering on-going DDoS attacks

#62
post #55

Earlier quoted context omitted.

OVH offers very comprehensive DDoS protection with all but the most budget servers: https://www.ovh.com/us/anti-ddos/

Comments from people seem to indicate their anti DDoS was "we blackhole you if you draw an attack". Has that changed?

Yes. This hasn’t been true for a long time. OVH runs a huge network and they invested in tools to mitigate and neutralize DDoS attacks[1]. Of course an attack can still saturate your servers’ NICs but they won’t drop you anymore.

At their scale they are basically forced to handle big attacks on regular basis. The fact that they offer this protection in their basic package is what makes them a great host[2].

They don’t get a fraction of the love Cloudflare gets, but they should.

[1] https://www.ovh.com/us/anti-ddos/hoovering-up.xml

[2] https://www.ovh.com/us/news/articles/a1171.protection-anti-d...

Re: Linode is suffering on-going DDoS attacks

#63
post #59
post #18

Earlier quoted context omitted.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)

I didn't downvote, but I can tell you almost certainly it was because of the snark.

Re: Linode is suffering on-going DDoS attacks

#64

I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't…

All of this +1000000

We've found that the more distributed the attack, the easier it is to defend. If it's truly distributed in nature, you can help spread the attack surface through things like anycasting if you have you multiple datacenters. Also, the more distributed it is, the more likely it is to be spread out among your different transit links. The less distributed, the more likely it will hit a single transit link and take that out. You might have 10 different transit providers with 10gig links. 100 gbps of transit is nothing to scoff at for small companies. You can have an 11 gbps DDoS that makes you unreachable for large numbers of people if that 11 gbps lands on a single link (due to AS path length, MED, preference, etc). People at your company are asking you "WTF network dude I thought we had 100gigs?" You can then try to mess with padding route advertisements, but that just messes more with legit traffic than attack traffic.

Same thing with aggregated ethernet links (LACP/802.3ad). You might be 10x10gigs with a single provider, giving you a "100gbps pipe" but if it comes from a few sources all going after multiple destinations, a single link in that bundle could easily be overwhelmed. Then you start playing with the hashing algorithm, but it never really solves the problem.

The only solution right now is to get a bigger boat and get 100gbps links. It's very expensive, not every transit provider offers that to every customer, lol if you run Cisco, and even then it's honestly not big enough. Overrunning 100gbps interfaces with DoS traffic isn't very hard to do if it comes from a few sources.

One must realize that not every country has the network diversity that the US does. In a lot of places, there are only a handful of AS networks that actually provide transit out of the country. This means that an entire country's network traffic can flow over a few (or often times one) providers. This turns into a giant laser that blows out single provider or single LACP member links. China is a great example of this. There are really only 2 AS that provide egress from China. The third is reserved for high ranking party officials and ultra, megarich people who pay the bribes. Korea (and all the broadband that HN people drool over) has a few more transit networks, but not many. Korea is a potent DDoS cannon.

As far as content goes it's by far mostly political and religious (> 70%). The remaining is the usual junk e-commerce, gambling, whistleblower sites, etc.

The lack of communication between transit providers and even between their customers is astoundingly bad. IMHO, there are good transit providers that will help you (NTT, Telia, XO) but most will tell you to go pound sand or not do anything until it starts disrupting their own network (Level3, Cogent, GTT, VZ, AT&T).

Re: Linode is suffering on-going DDoS attacks

#65

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet.

About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters.

Edit to be more specific: AWS gets hit with a lot of DDoS attacks. While all of AWS isn't unreachable during an attack, parts of it are. It's so large that you might not notice, but parts are unreachable. AWS/GCE size only makes it less noticeable, but they have no customer facing DDoS protection offerings. Their only offering is to buy more of their services. These providers don't have magical 1000000gbps links. They're regular 100gbps links (or 100gbps LACP channels) that can get overrun in large enough attacks.

Re: Linode is suffering on-going DDoS attacks

#66

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

What about DigitalOcean? Its pricing is comparable to Linode's, yet DigitalOcean is now the second largest web host in the world according to Netcraft. Do you still think that AWS, Google Cloud Platform, and Microsoft Azure offer better protection?

Here's the email I received (many times) when someone sent a smallish 1Gbit/s DDoS to my digitalocean server:

> Our system has automatically detected an inbound DDoS against your droplet named xyz with the following IP Address: xx.xx.xx.xx

> As a precautionary measure, we have temporarily disabled network traffic to your droplet to protect our network and other customers. Once the attack subsides, networking will be automatically reestablished to your droplet. The networking restriction is in place for three hours and then removed.

> Please note that we take this measure only as a last resort when other filtering, routing, and network configuration changes have not been effective in routing around the DDoS attack.

> Please let us know if there are any questions, we're happy to help.

This happened mere seconds after the DDoS begun! Therefore they lied about having tried to mitigate the attack.

No amount of contacting support got me un-blackholed before the 3 hour mark, and when I popped back into the network, I was blackholed again for another 3 hours...

I moved to a $3.50/mo OpenVZ VPS at OVH, and OVH's VAC system soaked up the DDoS just fine.

Re: Linode is suffering on-going DDoS attacks

#67
post #59
post #18

Earlier quoted context omitted.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)

I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it.

If you don't already have it enabled....well, good luck mate.

Re: Linode is suffering on-going DDoS attacks

#68

Yep, this started on Friday (Christmas day). I assume Linode itself is the target of the attacks, since they have spanned multiple regions.

Can confirm that the attacks started Friday. We got alarms for some tertiary services we run in their datacenters (multiple geos).

Re: Linode is suffering on-going DDoS attacks

#69

Looks like they have a history of suffering these kinds of attacks: (2012) Upcoming DDOS Attack - FINAL Warning - https://forum.linode.com/viewtopic.php?t=8530 (2013) Linode Mitigates DDoS Attack on Linode Manager - http://www.thewhir.com/web-hosting-news/linode-mitigates-ddo... (July 2015) Incident Report for Linode - http://status.linode.com/incidents/vnslh3rmm9gq So what makes them such an attractive target for th…

LOL at the "proof" attack. "We have a huge 500gbps cannon! But to show we're serious, here's a 2gbps attack. Take that!" That's hilarious. I wouldn't have taken it seriously. I also doubt that the group that threatened them is the same group that is attacking them. I bet it's at least 1 other group.

2gbps. I doubt that even registered on their bandwidth graphs.

Re: Linode is suffering on-going DDoS attacks

#70
post #67
post #59

Earlier quoted context omitted.

Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)

I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it. If you don't already have it enabled....well, good luck mate.

As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.
Post reply on HN