Linode is suffering on-going DDoS attacks • Hacker News Board

Re: Linode is suffering on-going DDoS attacks

#11

Post by swiley » Tue, Dec 29, 2015, 2:41 PM UTC

empressplay wrote: post #8

silverlight wrote: post #2

Been happening daily for about 5 days now. Pretty frustrating but I'm sure it's frustrating for them too. Thinking about moving to Google Compute Engine instead.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

Re: Linode is suffering on-going DDoS attacks

#12

Post by beachstartup » Tue, Dec 29, 2015, 2:46 PM UTC

empressplay wrote: post #8

silverlight wrote: post #2

Been happening daily for about 5 days now. Pretty frustrating but I'm sure it's frustrating for them too. Thinking about moving to Google Compute Engine instead.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

this is going to happen no matter what - the rate of consolidation in my opinion is actually accelerating. despite what anyone says, the vast majority of people choose their hosting provider in order to "not get fired", i.e. they are extremely risk-averse, to the point of irrationality, even when presented with massive savings, better service, better underlying hardware and network, and more transparency.

most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers).

at the end of the day it's a server sitting in a rack in a datacenter, connected to ethernet. beyond a certain level of quality (tier 3 dc, server class hardware, an enterprise quality network) it's really all the same. people should choose their hosting provider on 1. quality of implementation 2. price and 3. whether or not the provider actually gives a shit about you and your account, but they usually just go with the name, like many other markets.

Re: Linode is suffering on-going DDoS attacks

#13

Post by mmaunder » Tue, Dec 29, 2015, 2:51 PM UTC

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

Re: Linode is suffering on-going DDoS attacks

#14

Post by at-fates-hands » Tue, Dec 29, 2015, 2:52 PM UTC

Looks like they have a history of suffering these kinds of attacks:

(2012) Upcoming DDOS Attack - FINAL Warning - https://forum.linode.com/viewtopic.php?t=8530

(2013) Linode Mitigates DDoS Attack on Linode Manager - http://www.thewhir.com/web-hosting-news/linode-mitigates-ddo...

(July 2015) Incident Report for Linode - http://status.linode.com/incidents/vnslh3rmm9gq

So what makes them such an attractive target for these types of attacks?

Re: Linode is suffering on-going DDoS attacks

#15

Post by nothrabannosir » Tue, Dec 29, 2015, 2:53 PM UTC

[deleted] wrote: post #10

empressplay wrote: post #8

Earlier quoted context omitted.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

[deleted]

He(/she?) never said he wouldn't do it, just that it worried him. Back seat doesn't mean thrown out of the car.

People need to keep bringing this up, lest we completely forget about it and wake up when it's too late.

Re: Linode is suffering on-going DDoS attacks

#16

Post by yeukhon » Tue, Dec 29, 2015, 3:04 PM UTC

swiley wrote: post #11

empressplay wrote: post #8

Earlier quoted context omitted.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to use AWS, please build on multiple regions.

Re: Linode is suffering on-going DDoS attacks

#17

Post by alexandrerond » Tue, Dec 29, 2015, 3:05 PM UTC

mmaunder wrote: post #13

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

I am not sure there is a magic way to mitigate these attacks permanently other than playing cat and mouse until the attacker gets tired. Linode has in fact mitigated the attacks, only to find them pointing to a different part of their infra or DCs. At this point, it is probably a good sign that they haven't gone completely down (for long). Current tooling and botnets make DDoS-ing fairly easy today, despite countermeasures.

Re: Linode is suffering on-going DDoS attacks

#18

Post by kyrra » Tue, Dec 29, 2015, 3:06 PM UTC

mmaunder wrote: post #13

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

Re: Linode is suffering on-going DDoS attacks

#19

Post by yeukhon » Tue, Dec 29, 2015, 3:07 PM UTC

beachstartup wrote: post #12

empressplay wrote: post #8

Earlier quoted context omitted.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

this is going to happen no matter what - the rate of consolidation in my opinion is actually accelerating. despite what anyone says, the vast majority of people choose their hosting provider in order to "not get fired", i.e. they are extremely risk-averse, to the point of irrationality, even when presented with massive savings, better service, better underlying hardware and network, and more transparency. most of the…

> most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers).

Even AWS can charge you big time. As I am getting more and more familiar with AWS every day, the #1 thing on my list going forward is to to sit down with your TAM and organize architecture review. There are services on AWS lacking completeness and can bite you in the end if you go straight with it without knowing what you are getting into.

Disclaimer: I like AWS pretty well.

Re: Linode is suffering on-going DDoS attacks

#20

Post by mmaunder » Tue, Dec 29, 2015, 3:09 PM UTC

kyrra wrote: post #18

mmaunder wrote: post #13

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is the attack? Is it targeting a client or Linode itself? And if they're working with their upstream providers, what are they doing?

Edit: In the mean time, I am (sadly) moving docs.wordfence.com back into our data center. I have access to the Dallas server again and so am starting the transfer now. If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.