Earlier quoted context omitted.
So you're comparing DDoS of a provider with everyday spending? I was contracted a few months ago to save a website crumbling under its bills. They had $11k / month in AWS bills. Brought it down to $600 / month by switching them to Linode with a more reasonable stack. Not everything is a nail. Sometimes the mistake is choosing AWS.
If you reduced their bill from $11k to $600, I'm betting AWS was not the main problem.
Linode is suffering on-going DDoS attacks
151–160 of 186 posts
Re: Linode is suffering on-going DDoS attacks
#152Oh wow. I remember a couple months ago the ATL datacenter had network issues too. Really annoying, but I guess it's not their fault 100%. I wish they offered more DDoS protection solutions. I know some VPS companies specialize in that offering for things like game servers. It'd be nice if some sort of solution could just be included. I don't know if it's more of a technical issue or legal problem. As far as I know th…
OVH does VPS's and they have their own Anti-DDoS network setup that is pretty amazing: https://www.ovh.com/us/anti-ddos/
Re: Linode is suffering on-going DDoS attacks
#153Earlier quoted context omitted.
12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.
I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out…
If you're spending millions of dollars on making systems redundant, I completely understand. But that is not the same as spending millions of dollars to allow datacenters 4-7 to be on other vendors.
Maybe if you name some problems specific to multi-vendor support that took several man-months to fix I might comprehend. Just a sentence or two description. Thanks if you do, sorry for being confused if you don't.
Re: Linode is suffering on-going DDoS attacks
#154Earlier quoted context omitted.
Guilty. This has caused all kinds of pain for us this weekend. We use WPEngine to host some sites, who in turn host everything on Linode. Honestly WPEngine has some real nerve charging people big bucks for a failover plan that apparently doesn't exist. This is just another of a half-dozen or so Linode failures that took us and loads of other of their customers down completely. We're lucky that we planned for this ahe…
And yet after years on Linode they are still the best out of many I've tried the past 2 decades, weird how our experiences vary isn't it?
I guess what I should be really saying is that if you're hosting your platform on these services and offer some kind of redundancy, make that redundancy through another host.
Re: Linode is suffering on-going DDoS attacks
#155Earlier quoted context omitted.
I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out…
I know it's extremely hard to make a resilient system, but I don't understand how using multiple vendors makes it more complicated. Just on AWS you already have computers in different datacenters. How much harm can it really cause that datacenter 5 has 20% more ram on the nearly-identical machines, and you have to connect the network differently? If you're spending millions of dollars on making systems redundant, I c…
Anyway, the problem started with bad management to be honest, which I don't think my case is too rare to hear. When the first project rolled in, we ran PaaS on a single provider, but more projects joined, it was time to choose another vendor because of better equipment and better deals. Yet, none of them really provide good resiliency.
With the third vendor probably around 12-18 physical servers were purchased and managed by the vendor. It was openstack. That version of OpenStack at the time was not compatible with AWS, which later becomes our 4th vendor, and runs our dev environment. You want to run additional performance testing, well, we can't get the same elasticity as AWS because we only have so many physical servers to provision VMs, but that's where our real data lives. So we had to do our QA on AWS and doing the data copy can take a whole business day. The script we wrote for Amazon doesn't work on OpenStack. Security, network, I/O metrics are not consistent across vendors. That adds complexity to code. I know exactly how to abstract things, but just trust me :-) it really makes code hard to maintain, and really painful to integrate with multiple vendors. As a matter of fact, I don't like working with AWS API (with boto) myself because API response formats are inconsistent!
Finally, we got rid of one of them, then two and finally we are on the final stage of consolidating everything on AWS and focus on infrastructure automation and lower the number of chores.
Like many projects out there, things usually start out real nice, but then once you get too busy fighting fire here and there, you will accumulate some debts. If one is not careful, the debt can backfire and we had our lessons. There is no resiliency in most of the other vendors because they require you to purchase more servers and they themselves have hard time to go true elasticity. AWS, at least, for the most part, doesn't run out of instance availability that often (it happened a few times to our EMR processing). I do have some issues with AWS myself, but so far, AWS seems to be the only true cloud provider you can hang on to for several years.
Before building a grand multi-vendor infrastructure, build on a single-vendor well, then decide on the next step.
Re: Linode is suffering on-going DDoS attacks
#156Earlier quoted context omitted.
Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to us…
I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), b…
There is so much to with than just be able to spin up an VM and then run Ansible/Chef/Puppet on it. Heck I can write all of that in Fabric. There is no direct connect on Digital Ocean. I am not sure how you set up VPN with Digital Ocean or Linode. We use cloudformation on AWS, and I am pretty sure there is no such thing on Linode or Digital Ocean. Exception and response codes different across providers. Able to reproduce an environment from scratch is important to us, and of course, we try to do that in stages. I own a DO box myself, and that box turns out to be really slow in the NY region (where I live), maybe I am just an lucky bastard.
But to be honest, did you really build your entire infrastructure in three vendors to begin with? What are your reasons to really build on Linode, Digital Ocean and Vultr? How do you copy your data across environments? Are you splitting dev/qa/ci/sandbox/stage/prod?
Re: Linode is suffering on-going DDoS attacks
#157Earlier quoted context omitted.
Given my ignorance of much of these issues, I probably shouldn't be commenting (take my comment with a huge grain of salt). But the idea of depeering networks on the Internet for misapplication of a voluntary protocol seems like the beginning of the end of a free Internet (if ever such a thing existed). If BCP38 is critical to the success of the Internet, I think rather than ranting about those not implementing it, e…
Your heart's in the right place, but the Internet is built on policies of individual networks because there is nobody to enforce. Your suggestion back to me is simply mine in different clothing, because you think someone can enforce such a global requirement. Enforcing policy like "filter or get depeered" is the only way to achieve a global requirement like you want with the way the Internet is structured. As akerl p…
Re: Linode is suffering on-going DDoS attacks
#158Earlier quoted context omitted.
I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.
Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least commo…
Re: Linode is suffering on-going DDoS attacks
#159This may be a dumb question as I haven't read the bill, but isn't this what CISA is supposed to help guard against in the future? If Linode has an easier way of sharing info with the USG, can that help pinpoint and mitigate this attack from happening in the future? Edit: how about a response instead of downvotes?
Dealing with DDOS has always been way below anti-piracy on in the priority of the relevant US authorities. It's very easy to do and hard to identify those ultimately responsible. Although every now and again a big botnet command-and-control network gets shut down. Nobody's interested in the defensive side of ""cyberwarfare"".
Re: Linode is suffering on-going DDoS attacks
#160I would like to correlate the comments in this thread with past comments on every single article about AWS or GCE of the form "this is so expensive / complicated I run my boxes on Linode for half the price". DDoS protection is one of the things you pay for on the big clouds.
What DDoS protection does AWS provide? The only thing mentions on their webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection strategy is to open up your wallet. About 6 months ago they did hire Jeff from BlackLotus. Given that timeline, I'd expect them to announce some sort of DDoS protection offering in the next few quarters. Edit to be more specific: AWS gets hit with a lot of DDoS attacks.…
I've got no internal knowledge of how AWS operates, but I once went too far when load testing a new AWS machine (in terms of reqs/sec), and the traffic started getting dropped somewhere before reaching the box. Not sure if it was an elastic ip or behind an ELB, but I found it interesting.