Live data from Hacker News

Linode is suffering on-going DDoS attacks

status.linode.com

141–150 of 186 posts

Re: Linode is suffering on-going DDoS attacks

#141

Earlier quoted context omitted.

Your heart's in the right place, but the Internet is built on policies of individual networks because there is nobody to enforce. Your suggestion back to me is simply mine in different clothing, because you think someone can enforce such a global requirement. Enforcing policy like "filter or get depeered" is the only way to achieve a global requirement like you want with the way the Internet is structured. As akerl p…

Thanks for the kid-gloves reply :) I hadn't considered that there isn't really a central authority for controlling who runs a peer, aside from ICANN, but they have pretty loose reins. Funny that everyone waxes poetic about bitcoin being a revolution in anonymous and tacit network management. Meanwhile our little Internet experiment continues to be a HUGE tacit agreement to adhere to a handful of network protocols.

If you talk to ICANN, IANA, and friends, they're very clear (and careful to reiterate, even in minor threads) that all they do is run databases that contain interesting information. All of their policy revolves around admission to and management of said databases. It's comforting to think of authorities that govern the Internet, particularly when one of those organizations even has "Authority" in the name, but the truth is that the Internet is a miracle of decentralization as you say; we as network operators on the Internet give the databases meaning and force, not ICANN or any of its subordinates. ICANN has power because we have chosen their databases as the root of said power.

Even the fact that 'news.ycombinator.com' leads here is a de facto consensus since the Internet could, theoretically, switch roots tomorrow and completely invalidate every domain name. It won't happen, much to the chagrin of alt-root operators from the 90s and 2000s and contemporary attempts like NameCoin[0], but it can, and there is absolutely nothing ICANN could do about it. They charge admission to a well-maintained database that underpins this whole show, and that's pretty much the entirety of what they do.

It's on all of us to tend to the best interests of the Internet, and way too many people with access forget that. However, with responsibility like that also comes opportunity: once you realize that it's basically all of our good faith and de facto consensus holding this thing together, the barrier to entry for you to get involved with the Internet is suddenly far lower.

[0]: Oh yes, it's been tried, a lot: https://en.wikipedia.org/wiki/Alternative_DNS_root

Re: Linode is suffering on-going DDoS attacks

#142
post #126
post #116

If anyone from Linode admin team would like some help analyzing the attack/friendly advice on mitigations, please contact me (or anyone else at CloudFlare); we see a lot of these.

How do you guys trace the real packet sender of a packet with a spoofed IP address?

Clients that don't interpret javascript, typically.

Re: Linode is suffering on-going DDoS attacks

#143
post #140

Earlier quoted context omitted.

Actually "Amazon CloudFront also has filtering capabilities to ensure that only valid TCP connections and HTTP requests are made while dropping invalid requests. This takes the burden of handling invalid traffic (commonly used in UDP floods, SYN floods, and slow reads) off your origin." and "By using multiple PoPs, Amazon CloudFront has the inherent ability to help mitigate against both infrastructure and some applic…

Amazon Cloudfront is a CDN. A significant portion of a CDNs job is to be highly redundant and to handle DDoS. If you're comparing Amazon's offering to Linode, you should really compare to the protection Amazon offers on EC2 and similar VPS-like products.

I was merely answering the question "What DDoS protection does AWS provide?".

Normally you wouldn't expose a EC2 instance without putting cloudfront, elbs in front of it and locking down incoming traffic to cloudfront IPs.

Re: Linode is suffering on-going DDoS attacks

#144
post #126
post #116

If anyone from Linode admin team would like some help analyzing the attack/friendly advice on mitigations, please contact me (or anyone else at CloudFlare); we see a lot of these.

How do you guys trace the real packet sender of a packet with a spoofed IP address?

We don't, generally. It would be challenging.

If it were a volumetric attack, you could walk back links to find the source. But for anything but a huge attack which lasted for weeks/months, that would be inefficient use of resources.

Paul Vixie is really at the forefront of pushing for providers to solve this problem. Until that happens (and they've tried for a decade), it will remain technically difficult/impossible, so you need to use other solutions to mitigate attacks.

Re: Linode is suffering on-going DDoS attacks

#145
post #136

Earlier quoted context omitted.

Amusingly, we moved our infrastructure from Linode to AWS and cut our bill roughly in half about a year ago.

That can happen as well of course. Like I said, not everything is a nail. Sometimes AWS is the wrong choice, sometimes it's the right choice. However a lot of people jump on AWS when they really should not.

Agreed. We've got a moderately complex stack and on Linode the noisy neighbors problem caused us to require bigger boxes for certain layers than should really have been necessary.

Honestly OpsWorks on AWS has been a fucking dream from an ops perspective - we've been able to fully automate basically everything.

My only complaint about AWS is RDS: in order to squeeze maximum performance out of a database, you need to pay for IOPS, and boy do you pay. Fortunately, we only need a massive database instance a few weeks out of the year (our industry has some massive seasonal spikes), so it's not so bad.

Re: Linode is suffering on-going DDoS attacks

#146

I find this ironic because about 2 years ago I had a couple VMs with them that suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a whole other platform. Linode, on the other hand, simply blamed us for supposedly causing the repeat DDoS attacks - one after another for months on end. They even got rather flippant with me exclaiming how "dumb" I was to not understand that is was MY problem apparen…

Every point you make I can counter, I've been with them for years, have multiple VMs with them, find their support team fast, competent and courteous. DDOS is a problem for every ISP and every ISP customer.

Re: Linode is suffering on-going DDoS attacks

#147

Earlier quoted context omitted.

Thank you for being so understanding!

Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point. I filed a support ticket trying to get more information about the outage, and the response I got was absolutely worthless. No part of this has made me feel better about Linode at all. AWS is so massive that they can just sustain most DDOSes, and they write real postmortems a…

>> Assigning fault isn't productive. But as Linode customers, what are we to do? My trust in Linode's reliability is completely shot at this point.

Then it will also be the same when you are at XYX ISP and they get DDOS'ed. This isn't a Linode specific problem.

>> AWS is so massive that they can just sustain most DDOSes

No.

http://stackoverflow.com/questions/13244713/does-aws-protect...

"It will not, however, employ security algorithms needed to deflect application layer attacks (e.g., HTTP flood) nor will it help against larger network layer threats."

https://www.cloudflare.com/ddos/

"Layer 3 and 4 attacks are difficult—if not impossible—to mitigate with an on-premise solution. "

Re: Linode is suffering on-going DDoS attacks

#148
post #65

Earlier quoted context omitted.

What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet. About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters. Edit to be more specific: AWS gets hit with a lot of DDoS attacks.…

I don't know, but all traffic to GCE is routed through Google's frontend, which provides in-built DDoS protections.

So...Google can stop all DDOS's? Everyone should dump AWS, Linode, Cloudflare, etc. and go to Google for their VMs then.

Re: Linode is suffering on-going DDoS attacks

#149

Earlier quoted context omitted.

That is true. However, how many of the people impacted by the current DDoS against Linode are only affected BECAUSE they are using Linode?

Guilty. This has caused all kinds of pain for us this weekend. We use WPEngine to host some sites, who in turn host everything on Linode. Honestly WPEngine has some real nerve charging people big bucks for a failover plan that apparently doesn't exist. This is just another of a half-dozen or so Linode failures that took us and loads of other of their customers down completely. We're lucky that we planned for this ahe…

And yet after years on Linode they are still the best out of many I've tried the past 2 decades, weird how our experiences vary isn't it?

Re: Linode is suffering on-going DDoS attacks

#150

Looks like they have a history of suffering these kinds of attacks: (2012) Upcoming DDOS Attack - FINAL Warning - https://forum.linode.com/viewtopic.php?t=8530 (2013) Linode Mitigates DDoS Attack on Linode Manager - http://www.thewhir.com/web-hosting-news/linode-mitigates-ddo... (July 2015) Incident Report for Linode - http://status.linode.com/incidents/vnslh3rmm9gq So what makes them such an attractive target for th…

And Google and AWS and Cloudflare and DigitalOcean and every other VPS provider.
Post reply on HN