Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

Post reply on HN
Print view View original

71–80 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#71

Post by scurvy »

thrownaway2424 wrote: post #37

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

After thinking about it more, I now realize that there is one way that AWS/GCE can lessen effects of a DDoS. Given that a lot of attack machines are hosted inside of AWS/GCE, they can handle this with lots of internal network bandwidth. They see less north/south (transit) attack bandwidth than a regular hoster would.

In short, if a large amount of the attack traffic originates inside of AWS/GCE, it's better to be off in AWS/GCE.

Between the two, neither have any kind of automated tooling to detect and shutdown rogue attack instances (AFAICT). They still rely on third parties to tell them "Hey, you're sending me 300 gbps of DoS traffic."

Re: Linode is suffering on-going DDoS attacks

#72

Post by gist »

scurvy wrote: post #65
thrownaway2424 wrote: post #37

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet. About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters. Edit to be more specific: AWS gets hit with a lot of DDoS attacks.…

"I'd expect them to announce some sort of DDoS protection offering in the next few quarters."

It will be very interesting to see how cost effective that protection actually is and what they charge for it. Nice profit center off of FUD.

Re: Linode is suffering on-going DDoS attacks

#74

Post by jsmthrowaway »

devicenull wrote: post #60
gtrubetskoy wrote: post #51

I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't…

> The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. These days it's the hundreds of thousands of misconfigured NTP servers, recursive DNS servers, and various other protocols being abused for reflection attacks. Granted, it still requires that the attacker have the ability to…

BCP 38 is like herd immunity with immunization, and much like anti-vaccine folks, networks that don't follow it are knowingly choosing to infect people for any number of reasons. Despite your claim, it is extremely easy to implement and has been a known best practice, with accompanying educational Web sites devoted to the topic[0], for many years. There are nearly zero reasons for your AS to transmit forged packets, and if your configuration allows it, you are knowingly making the Internet a worse place with your laziness and transferring your laziness to other people like me that carry pagers. This isn't a surprise to any network administrator unless they've spent their entire career not reading RFCs.

I'm of the opinion that networks that allow customers to emit forged source addresses should be depeered until they take the literal hour to fix it. "But we have to update equipment when we get new blocks!" Boo hoo. Automate it or get off the Internet so I can stop spending my life dealing with your customer's amplified traffic.

If you run a non-transit/eyeball AS, you are in the absolute best position to stop these types of attacks from ever happening. The rest of the Internet, particularly your transit peers, can't really clean up after you on this one. Do us all a favor.

[0]: http://bcp38.info

Re: Linode is suffering on-going DDoS attacks

#75

Post by scurvy »

mwcampbell wrote: post #24
yeukhon wrote: post #16

Earlier quoted context omitted.

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to us…

I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.

Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least common denominator amongst cloud providers, you've lost. To really succeed, you must choose the best cloud offering and then use everything that they offer. Otherwise, why are you running stuff in the cloud anyway?

Also, steer clear of "cloud neutral" services and products that will magically move data and services across cloud providers. Interop is the last thing on any proprietary vendor's mind. An example. You have a pair of border edge routers. Do you buy 2 Junipers, 2 Ciscos, or 1 Juniper and 1 Cisco for fear of a bad vendor bug taking out all the routers? I'll tell you which one I would not choose. The .com TLD nameserver requirements used to mandate dual vendor setups. They sure did learn their lesson.

Re: Linode is suffering on-going DDoS attacks

#76

Post by Keverw »

Oh wow. I remember a couple months ago the ATL datacenter had network issues too. Really annoying, but I guess it's not their fault 100%. I wish they offered more DDoS protection solutions. I know some VPS companies specialize in that offering for things like game servers. It'd be nice if some sort of solution could just be included. I don't know if it's more of a technical issue or legal problem. As far as I know the only way is to null route(which is an inconvenience to the website, but no traffic going to the router other client's are on) or just adding a bunch of servers to "soak" up the extra bandwidth.

Re: Linode is suffering on-going DDoS attacks

#77

Post by scurvy »

jsmthrowaway wrote: post #70
scurvy wrote: post #67

Earlier quoted context omitted.

I didn't downvote, but there's one thing to keep in mind. You must request RTBH functionality with every peer I've ever worked with. It's free, but they don't set it up automatically. You need to request it during provisioning or wait 3-5 days for someone to activate it. If you don't already have it enabled....well, good luck mate.

As of a few years ago Linode got transit from the facilities they are in and almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would be incredibly foolish to operate a hosting provider without it.

Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there.

From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?

Re: Linode is suffering on-going DDoS attacks

#78

Post by scurvy »

gist wrote: post #72
scurvy wrote: post #65

Earlier quoted context omitted.

What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet. About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters. Edit to be more specific: AWS gets hit with a lot of DDoS attacks.…

"I'd expect them to announce some sort of DDoS protection offering in the next few quarters." It will be very interesting to see how cost effective that protection actually is and what they charge for it. Nice profit center off of FUD.

If they price it like cheap insurance, they will make a lot of money and offer a worthwhile service. Pay them a low monthly rate (X) to have the service active and then pay hourly at Y rate during an attack. You can play with X and Y to maximize your monthly revenue while still offering a cheap (but very valuable) service that retains your customers.

Re: Linode is suffering on-going DDoS attacks

#79

Post by antsar »

lftl wrote: post #27
mmaunder wrote: post #20

Earlier quoted context omitted.

That's how it's done. We use colo and have a DC outside Seattle with 5 uplinks to Tier 1 providers. We suffered a 20 Gbps (which is relatively small) on a 1 Gbps port that completely saturated our uplink. Our data center worked with their upstreams to route the traffic through a layer 7 DDoS mitigation service. It was amazingly effective. So I'm curious about what they're going through - I want details. How big is th…

If I can get some clarity on who did this, why and how it got fixed and why it won't happen in future I'm sure me and a bunch of other frustrated ops folks will consider sticking with them or moving back. I've been a generally happy Linode customer for a long time, but this is my biggest gripe with them. I can almost guarantee you'll never get this. Linode just doesn't do it. Their postmortems on issues (if they ever…

Same experience here. I asked Support once about an ongoing outage and was told that "there were some network related issues". Really? I wouldn't have guessed...

Re: Linode is suffering on-going DDoS attacks

#80

Post by frik »

thrownaway2424 wrote: post #37

I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.

you mean discussions like this (2 days ago), first comment sub-thread: https://news.ycombinator.com/item?id=10794951
Post reply on HN