Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

21–30 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#21
post #11

Earlier quoted context omitted.

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

How does one set up to use multiple vendors?

Re: Linode is suffering on-going DDoS attacks

#22
post #11

Earlier quoted context omitted.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

How does one set up to use multiple vendors?

You buy servers from multiple different companies, you install software on them.

Re: Linode is suffering on-going DDoS attacks

#23

Looks like they have a history of suffering these kinds of attacks: (2012) Upcoming DDOS Attack - FINAL Warning - https://forum.linode.com/viewtopic.php?t=8530 (2013) Linode Mitigates DDoS Attack on Linode Manager - http://www.thewhir.com/web-hosting-news/linode-mitigates-ddo... (July 2015) Incident Report for Linode - http://status.linode.com/incidents/vnslh3rmm9gq So what makes them such an attractive target for th…

The fact that so many people on their IRC channel will react to the attacks.

Re: Linode is suffering on-going DDoS attacks

#24
post #16
post #11

Earlier quoted context omitted.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to us…

I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.

Re: Linode is suffering on-going DDoS attacks

#25

So weird that Linode hasn't been able to mitigate this. I'd love to learn more about what's happening there. Particularly since we host our production documentation site on a linode vm. I want to move it off their server and into our DC but can't access the server to do it. Bummer. I've been such a big fan of theirs.

I've been with Linode for years and they've fought off more than a few DDOS attacks in the past, so they certainly are competent in that regard. I think this one must be a higher level of severity than previous ones. Generally, they seem ok with this stuff. I don't think there are any DDOS-proof VPS's out there.

Re: Linode is suffering on-going DDoS attacks

#26
post #22

Earlier quoted context omitted.

How does one set up to use multiple vendors?

You buy servers from multiple different companies, you install software on them.

Anyone can do that. I was actually wondering if there are software to automate this.

Re: Linode is suffering on-going DDoS attacks

#27
post #18

Earlier quoted context omitted.

If the DoS is large enough there is little they can do if their downstream is 100% saturated. They would either need more capacity or for their upstream providers to filter the attack for them.

That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is th…

If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back.

I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever come) are light on details of the problem, and even lighter on details about their solutions. They seem to do a decent job of addressing most issues, because they rarely have recurring problems, but they never really talk about what they're doing.

Re: Linode is suffering on-going DDoS attacks

#28

Yep, this started on Friday (Christmas day). I assume Linode itself is the target of the attacks, since they have spanned multiple regions.

Dallas is their oldest DC, so they have the most customers there and therefore the most attack surface area at that location. It could just as easily be one of their customers.

Re: Linode is suffering on-going DDoS attacks

#29
post #11

Earlier quoted context omitted.

The best thing for reliability is to use multiple companies. AWS has proved that multiple times.

How does one set up to use multiple vendors?

You might consider taking a look at http://www.cloud66.com/

Re: Linode is suffering on-going DDoS attacks

#30
post #16

Earlier quoted context omitted.

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to us…

I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.

Human capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also.

There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for tax purpose). Small to medium usually don't have time to deal with multiple vendors. Try to be vendor agnostic is really great but again, even in the world of open stack, you either manage all of your open stack installation yourself (which is expensive), or you end up one vendor running one version behind, or not offering 100% feature.

I think it is more worthwhile to first complete automation and self-healing in one vendor, before thinking leaping to another one. It took so many engineers at Netflix to build a resilient AWS environments. VMs go down all the time, more often than AWS pushing some bad commits and cause half of their native services go down. There is so much more to engineer in other areas and those are more likely to put you out of service first, so unless you want to all your team dealing with fires every day.... we were putting band-aids together most of the time.

Post reply on HN