Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

Post reply on HN
Print view View original

171–180 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#171

Post by scurvy »

X-Istence wrote: post #93
scurvy wrote: post #77

Earlier quoted context omitted.

Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there. From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?

Just because they purchase transit doesn't mean they don't have their own AS. Everyone has to purchase transit at some point. Transit != AS.

Where did I say that purchasing transit relates to an AS? I was speaking out them separately. "House bandwidth" in most colos is pretty garbage, and you don't have much control over it.

I'm a lot less impressed with Linode after this thread than I was before.

Re: Linode is suffering on-going DDoS attacks

#172

Post by scurvy »

jsmthrowaway wrote: post #134
scurvy wrote: post #77

Earlier quoted context omitted.

Wait, you're saying that Linode uses facility transit? Like, they buy bandwidth from Savvis and TelX? Well that would be the problem right there. From what I can tell, Linode doesn't even have their own AS for customer traffic? It appears that they have an AS for some internal use, but not for customers?

I don't know if I'd outright call it a problem, per se, since in a couple facilities (say, Fremont) it's a good thing given how good the provider (say, Hurricane Electric) is at providing transit. So in the datacenters where the provider is better at transit than facilities (say, Fremont), the equation breaks in their favor. When I left they also didn't really have the staff to run a proper NOC for a full-on AS; one…

Real talk: it's not that hard to run your own network. It sounds like Linode was skirting by on not making the infrastructure investment both in hardware and people. It's finally come back to bite them. No excuses and no pity for them.

Also, I wouldn't run around saying that having HE in house is a great thing.

Re: Linode is suffering on-going DDoS attacks

#173

Post by mark-ruwt »

mwcampbell wrote: post #165
mark-ruwt wrote: post #164

Earlier quoted context omitted.

No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from sc…

I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.

Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.

Re: Linode is suffering on-going DDoS attacks

#174

Post by mark-ruwt »

yeukhon wrote: post #156
mark-ruwt wrote: post #110

Earlier quoted context omitted.

I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), b…

Maybe, maybe not. I still believe in single-vendor approach, perphas because in my view I am either going for AWS or GCE. There is so much to with than just be able to spin up an VM and then run Ansible/Chef/Puppet on it. Heck I can write all of that in Fabric. There is no direct connect on Digital Ocean. I am not sure how you set up VPN with Digital Ocean or Linode. We use cloudformation on AWS, and I am pretty sure…

No, not at all! http://areyouwatchingthis.com is almost 10 years old at this point, and the architecture in its current form wasn't solidified until last year. As a one-man shop, redundancy and failover are my best friends.

Re: Linode is suffering on-going DDoS attacks

#175

Post by scurvy »

thrownaway2424 wrote: post #107
scurvy wrote: post #91

Earlier quoted context omitted.

I'd imagine they use VRF's to quickly segment the traffic after ingress. Google.com might have DDoS protection, but I'm wary that it extends to GCE. I've read about Google Andromeda, but there's no real meat in any article about DDoS mitigation.

This document specifically claims that "All traffic is routed through custom GFE (Google Front End) servers to detect and stop malicious requests and Distributed Denial of Service (DDoS) attacks." https://cloud.google.com/security/whitepaper

How though? There's remarkably little information in there for being a whitepaper. If all they do is drop Christmas-tree packets and bogus UDP/ICMP traffic, that's not much in the way of protection. I'd like to see exactly what/how they're doing. How do they know what traffic is malicious? Do you get a control panel to block certain L7 traffic? What L7 inspectors do they support?

Sorry, but I'm not buying it.

Re: Linode is suffering on-going DDoS attacks

#176

Post by USNetizen »

mordocai wrote: post #168
USNetizen wrote: post #162

Earlier quoted context omitted.

Not so fast. We migrated all of our former Linode clients to AWS and Azure and have not suffered a single DDoS in the roughly 2 years since the move to other providers. At Linode, on the other hand, we suffered more than a dozen in less than a year - even with different VMs, different IPs, different OSes and different configurations. We were also plagued with overwhelming attempts to brute-force SSH and other service…

To be fair, i've had a single linode VPS for years (i'd have to check to figure out how many) and this ddos is the first time i've ever had any issues whatsoever with my linode.

Luck of the draw. Certain IP blocks in the Linode range are attacked all the time, as evidenced here. We had a couple VMs that were never attacked, but far too many of them were on a regular basis for us to even consider staying with Linode.

Re: Linode is suffering on-going DDoS attacks

#177

Post by ryukafalz »

huac wrote: post #124
USNetizen wrote: post #111

I find this ironic because about 2 years ago I had a couple VMs with them that suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a whole other platform. Linode, on the other hand, simply blamed us for supposedly causing the repeat DDoS attacks - one after another for months on end. They even got rather flippant with me exclaiming how "dumb" I was to not understand that is was MY problem apparen…

+1 for IP block DDOS'ing - last time I ran a Linode VM it was DDOS'd to smitheereens within half an hour of putting it up (and of course, without any notice that I was liable for overages...)

>(and of course, without any notice that I was liable for overages...)

Inbound traffic does not count against your bandwidth quota, so if you were the target of a DoS attack, this would not result in overages.

https://www.linode.com/pricing

(Unless you were responding to all that traffic, that is.)

Re: Linode is suffering on-going DDoS attacks

#178

Post by jafingi »

Linode is _still_ hit by DoS. They currently have a "Major Outage" in the London DC. Seems like there are issues in almost all DC's.

Unbelievable they can't get that DoS stopped! My server have had 19 outages the past 7 days, and over 5 hours of downtime!

Still no e-mail from Linode whatsoever..

Re: Linode is suffering on-going DDoS attacks

#179

Post by kingrolo »

workitout wrote: post #146
USNetizen wrote: post #111

I find this ironic because about 2 years ago I had a couple VMs with them that suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a whole other platform. Linode, on the other hand, simply blamed us for supposedly causing the repeat DDoS attacks - one after another for months on end. They even got rather flippant with me exclaiming how "dumb" I was to not understand that is was MY problem apparen…

Every point you make I can counter, I've been with them for years, have multiple VMs with them, find their support team fast, competent and courteous. DDOS is a problem for every ISP and every ISP customer.

I've probably had some 50 nodes with Linode over the last 7ish years. Until about 2 years ago I would enthusiastically recommend them to anyone who would listen, but I've found the service has slipped massivel over the last couple of years and now we're trying to migrate people away. We have about 10 nodes left now and blips of lost connectivity and hardware failures are common unfortunately, and the feeling I get from their support team is that this is just to be expected now.

Re: Linode is suffering on-going DDoS attacks

#180

Post by pbowyer »

jafingi wrote: post #178

Linode is _still_ hit by DoS. They currently have a "Major Outage" in the London DC. Seems like there are issues in almost all DC's. Unbelievable they can't get that DoS stopped! My server have had 19 outages the past 7 days, and over 5 hours of downtime! Still no e-mail from Linode whatsoever..

I know. I didn't mind when the first outage happened in London (24th?) - these things happen. But now... this is getting beyond a joke. I'm going to have to consider moving, which as I host for friends is not as straightforward as it should be (since they control the DNS)
Post reply on HN