Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

51–60 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#51
I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever.

The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks.

Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't as distributed as it may seem - it can just be a couple of machines on a very well connected network (e.g. a university). But getting a hold of someone in the middle to filter that traffic can take a very long time or be outright impossible. First-responder network engineers (typically referred to as "security") are overworked and underqualified, and the people who really know their stuff typically can't be bothered with silly DOS attacks.

We've also observed that (D)DOS's happen because of content. Anything political, religious, or whatever other shade of the many things someone out there disapproves of is a potential target for a DOS. Contrary to what you may read in the press, extortion is only a small minority of all DOS attacks out there. We've actually told customers to go away because their content was too DOS-prone.

And because these things usually happen across countries, even though they are very real crimes that cause serious damage and cost money, they are hardly ever prosecuted. As the target of a DOS all you want is for it to stop, nobody ever bothers reporting it to the authorities afterwards (because how would you even know who the "authority" is).

Re: Linode is suffering on-going DDoS attacks

#52

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

As a Linode customer, I guess this wouldn't have happened with AWS, but the way they handle it is still far better than "your server, your bandwidth, deal with it" that you'd get with e.g. Hetzner dedicated servers. We had several hours of downtime, which sucks, but other than that we didn't have to worry about taking measures against the DDoS ourselves.

Re: Linode is suffering on-going DDoS attacks

#53
post #30

Earlier quoted context omitted.

Human capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also. There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for ta…

12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.

I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out fire, people rarely worked eight hours a day, we were all consultants.

Re: Linode is suffering on-going DDoS attacks

#54

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

On AWS the DDoS hits the wallet instead. What’s the difference?

It’s “denial of service,” not denial of server and network resources.

Re: Linode is suffering on-going DDoS attacks

#55

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

What about DigitalOcean? Its pricing is comparable to Linode's, yet DigitalOcean is now the second largest web host in the world according to Netcraft. Do you still think that AWS, Google Cloud Platform, and Microsoft Azure offer better protection?

OVH offers very comprehensive DDoS protection with all but the most budget servers:

https://www.ovh.com/us/anti-ddos/

Re: Linode is suffering on-going DDoS attacks

#56

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

That is true. However, how many of the people impacted by the current DDoS against Linode are only affected BECAUSE they are using Linode?

Guilty.

This has caused all kinds of pain for us this weekend. We use WPEngine to host some sites, who in turn host everything on Linode.

Honestly WPEngine has some real nerve charging people big bucks for a failover plan that apparently doesn't exist. This is just another of a half-dozen or so Linode failures that took us and loads of other of their customers down completely. We're lucky that we planned for this ahead of time, but we weren't 100% ready to go live on a competing service either. A lot of folks are working on their vacations right now.

I think the two questions I'll be asking every host now and into the future are:

1) Do you host your services on Linode. 2) If you do, do you failover to another provider?

A yes to the first question and a no to the second is a non-starter in my experience.

Re: Linode is suffering on-going DDoS attacks

#57
post #34

Just 2 days ago, a Linode employee was badmouthing AWS here on HN for being too expensive: https://news.ycombinator.com/item?id=10796094 A DDoS will be much more expensive to customers than choosing AWS over Linode (or an equivalent low-priced service). EC2 has been around since 2006, and never has had any issues resembling this.

So you're comparing DDoS of a provider with everyday spending? I was contracted a few months ago to save a website crumbling under its bills. They had $11k / month in AWS bills. Brought it down to $600 / month by switching them to Linode with a more reasonable stack. Not everything is a nail. Sometimes the mistake is choosing AWS.

If you reduced their bill from $11k to $600, I'm betting AWS was not the main problem.

Re: Linode is suffering on-going DDoS attacks

#58
post #55

Earlier quoted context omitted.

What about DigitalOcean? Its pricing is comparable to Linode's, yet DigitalOcean is now the second largest web host in the world according to Netcraft. Do you still think that AWS, Google Cloud Platform, and Microsoft Azure offer better protection?

OVH offers very comprehensive DDoS protection with all but the most budget servers: https://www.ovh.com/us/anti-ddos/

Comments from people seem to indicate their anti DDoS was "we blackhole you if you draw an attack". Has that changed?

Re: Linode is suffering on-going DDoS attacks

#59
post #18

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

Heard of nullroutes?

(Edit: how is it that perfectly valid technical solutions get downvoted?)

Re: Linode is suffering on-going DDoS attacks

#60

I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't…

> The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks.

These days it's the hundreds of thousands of misconfigured NTP servers, recursive DNS servers, and various other protocols being abused for reflection attacks.

Granted, it still requires that the attacker have the ability to spoof packets, but preventing that requires even more time investment and has very little benefit to the ISP.

Post reply on HN