Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

Post reply on HN
Print view View original

161–170 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#161

Post by mmaunder »

ryanlol wrote: post #59
kyrra wrote: post #18

Earlier quoted context omitted.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

Heard of nullroutes? (Edit: how is it that perfectly valid technical solutions get downvoted?)

Attackers shift to a new target on the same net as IP's are null routed, until the entire network has null routed itself which achieves the same result as DDoS'ing the network. So I'm not sure I agree that removing target IP's from your network is a great solution.

Re: Linode is suffering on-going DDoS attacks

#162

Post by USNetizen »

workitout wrote: post #146
USNetizen wrote: post #111

I find this ironic because about 2 years ago I had a couple VMs with them that suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a whole other platform. Linode, on the other hand, simply blamed us for supposedly causing the repeat DDoS attacks - one after another for months on end. They even got rather flippant with me exclaiming how "dumb" I was to not understand that is was MY problem apparen…

Every point you make I can counter, I've been with them for years, have multiple VMs with them, find their support team fast, competent and courteous. DDOS is a problem for every ISP and every ISP customer.

Not so fast. We migrated all of our former Linode clients to AWS and Azure and have not suffered a single DDoS in the roughly 2 years since the move to other providers. At Linode, on the other hand, we suffered more than a dozen in less than a year - even with different VMs, different IPs, different OSes and different configurations. We were also plagued with overwhelming attempts to brute-force SSH and other services from IPs in China while we hosted with Linode. Nowhere else did we see so much malicious activity targeted against our clients' VMs as we did at Linode.

During that time, Linode support tried ruthlessly to pass the buck and blame us and our clients, denying they had any major problems. Well, here we are and Linode DDoS attacks are STILL occurring on a regular basis, still making news, and they seem to do little about it in terms of a long-term solution. Whereas I have yet to have a single client suffer a Linode-scale DDoS on AWS or Azure at all after almost a half decade of using them.

Re: Linode is suffering on-going DDoS attacks

#163

Post by davidvwp »

busterarm wrote: post #56
Diederich wrote: post #50

Earlier quoted context omitted.

That is true. However, how many of the people impacted by the current DDoS against Linode are only affected BECAUSE they are using Linode?

Guilty. This has caused all kinds of pain for us this weekend. We use WPEngine to host some sites, who in turn host everything on Linode. Honestly WPEngine has some real nerve charging people big bucks for a failover plan that apparently doesn't exist. This is just another of a half-dozen or so Linode failures that took us and loads of other of their customers down completely. We're lucky that we planned for this ahe…

Hey Busterarm,

David here from WPE. Are you using our HA solution (Geographic redundancy)? Did that fail? If so, did you open a ticket with support & inquire about an SLA credit?

Keep in mind we have many levels of redundancy with all of our plans, but not every plan includes Geo redundancy. Very few sites anywhere truly use hot/hot geo redundancy because of the complexity of database syncing and the expense of duplicating server environments in different data centers.

In a DDoS attack (which can happen at any data center, backbone provider, etc.) or any other data center wide outage, the only work around is Geo redundancy. Many hosts have different infrastructure providers and offsite redundancy (e.g. all of our customers enjoy offsite backups), which can allow you to recover your site even if the data center burns down, but don't necessarily provide a true hot/hot level of redundancy.

A true hot/hot configuration requires hardware in multiple locations, live database syncing, and geo load balancing. While some of our customers do purchase geo redundancy, it isn't a default part of every account.

Again, if you're an HA customer and that failed, please open a ticket. If you have questions on Geo redundancy, you can ask about it in that ticket as well. If you do move hosts, remember any host can be the victim of a DDoS attack and unless you have true hot/hot or hot/warm geo redundancy with your account, you could still be susceptible to data center wide outages like large-scale DDoS attacks.

-D

Re: Linode is suffering on-going DDoS attacks

#164

Post by mark-ruwt »

mwcampbell wrote: post #118
mark-ruwt wrote: post #110

Earlier quoted context omitted.

I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), b…

Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.

No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps.

Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated.

Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.

Re: Linode is suffering on-going DDoS attacks

#165

Post by mwcampbell »

mark-ruwt wrote: post #164
mwcampbell wrote: post #118

Earlier quoted context omitted.

Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.

No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from sc…

I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.

Re: Linode is suffering on-going DDoS attacks

#166

Post by technikempire »

Whenever I see top level comments where people are saying "this is what you get for not going with X", I am more and more convinced that a competitor is doing this. I for one am not leaving Linode. I JUST recommended to several new clients to setup at Linode and I've already had to explain that server setup is halted because of this attack. Did I recommend moving away from Linode? Nope. Did I hang my head in shame or stutter when I delivered the news, feeling it would reflect poorly on me or Linode? Nope. I told my clients this is the best place for them and I stick by it. Just because there are scumbags specifically attacking your service right now, doesn't mean another service is better, or that your service is poor. It just means that hey, news flash, there are scumbags in the world.

Re: Linode is suffering on-going DDoS attacks

#167

Post by dangson »

mwcampbell wrote: post #3

Yep, this started on Friday (Christmas day). I assume Linode itself is the target of the attacks, since they have spanned multiple regions.

This helped me a bit because most of my clients are on vacation so I haven't received as many emails about sites being down/slow as I normally would've.

Re: Linode is suffering on-going DDoS attacks

#168

Post by mordocai »

USNetizen wrote: post #162
workitout wrote: post #146

Earlier quoted context omitted.

Every point you make I can counter, I've been with them for years, have multiple VMs with them, find their support team fast, competent and courteous. DDOS is a problem for every ISP and every ISP customer.

Not so fast. We migrated all of our former Linode clients to AWS and Azure and have not suffered a single DDoS in the roughly 2 years since the move to other providers. At Linode, on the other hand, we suffered more than a dozen in less than a year - even with different VMs, different IPs, different OSes and different configurations. We were also plagued with overwhelming attempts to brute-force SSH and other service…

To be fair, i've had a single linode VPS for years (i'd have to check to figure out how many) and this ddos is the first time i've ever had any issues whatsoever with my linode.

Re: Linode is suffering on-going DDoS attacks

#169

Post by scurvy »

X-Istence wrote: post #97
scurvy wrote: post #75

Earlier quoted context omitted.

Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least commo…

Comcast as a company purchases from two vendors to avoid lock-in and to get the best prices. Juniper and Cisco, although last I heard Arista was also starting to be used. Dual-sourcing makes sense in a lot of cases.

You can definitely dual source for separate parts of your network. Just don't dual source for the component in the same deployment. You do not want to run a Juniper and Cisco router in a pair, both talking with each other running IS-IS, VRRP, etc.

Re: Linode is suffering on-going DDoS attacks

#170

Post by scurvy »

fanf2 wrote: post #158
scurvy wrote: post #75

Earlier quoted context omitted.

Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least commo…

I thought the .com servers had been running Verisign's proprietary Atlas software since almost forever - 15 years?

I was referring to routers. They have to buy routers from someone.
Post reply on HN