121–130 of 151 posts
Earlier quoted context omitted.
We stuck with Linode after that, but the important parts now failover to Vultr.
We stuck with Linode after last Christmas too, but this new attack looks like the final straw. How have you found Vultr?
I don't get the hate towards linode here, on hacker news. I've been their client for a couple of years now and I find it an excellent vps provider. Excellent uptime and performance at a pretty good price. AWS has a few outages every year. Google just had one last week. Azure sucks balls. So, why the hate? Is it because it competes with some ycombinator startups?
But could it simply be severity of the attacks? I keep seeing comments about a 2 week ddos attack last christmas - that's something that i would be shocked to see Google/AWS succumb to. Not that Google/AWS attacks don't happen, i just can't imagine them being down for ~2weeks
(I imagine it was just one datacenter from Linode, not the entire service, fwiw)
I don't get the hate towards linode here, on hacker news. I've been their client for a couple of years now and I find it an excellent vps provider. Excellent uptime and performance at a pretty good price. AWS has a few outages every year. Google just had one last week. Azure sucks balls. So, why the hate? Is it because it competes with some ycombinator startups?
Linode has a bunch of great features, but after seeing them get hacked a half dozen times over really silly things, more DDOS's than you would be happy with, and frankly I have had interactions with their management (just online) and was sorry to have had said interactions.
You can also read a bunch of implications from former employees about their management, but you can feel free to discount that given how many times ex-employees are a bit pissed.
- Former Linode Customer
Earlier quoted context omitted.
Google and AWS probably have sophisticated DDOS mitigation (can anyone comment on this?) and you can scale up pretty quickly. But if your service is the direct target of the attack (as opposed to the whole provider) and your servers are getting hammered... Even if you architecture allows quick horizontal scaling, you still face a tough decision. The attack could go on for days, and the hosting costs can go really hig…
Attacks are rarely targeted to the hosting providers. They usually target a specific customer. Google/AWS probably have 100 times the capacity (and redundancy and architecture reliability and failover and awesomeness) of linode. That means that, first, they can't be put down easily, second, a DDoS is limited to a small subset of the infrastructure and doesn't bleed to every customers and services. As for traditional…
OVH hasn't been doing this for a while, they got some beefy ddos protection setup for this exact reason - it was way too easy to take down someone for hours
Hetzner (another big european hosting provider) followed recently: https://news.ycombinator.com/item?id=12403783
Online.net also has included protection (+ paid upgrades)
At least here in europe the big hosting providers are all switching to providing included protection for all their customers, at least for traffic intensive attacks which hurt everyone
Earlier quoted context omitted.
For our setup we just have master-master replication across two different hosting providers.
I've been wondering about this lately. Is it really feasible for a small (one man?) team to keep master-master MySQL replication over WAN running smoothly?
If you use mysql's read_only flag and application users don't have SUPER permissions, you can easily prevent writes to the wrong server; set read_only = 1 in my.cnf and manually set it to 0 on exactly one of the masters. Use the read_only flag to drive automation for which server to send writes to.
Manual failover is set old server read_only, kill existing connections (read_only flag is cached), wait for replication to catch up, set new server read_only = 0. You can make a script to do this with one button, but I wouldn't recommend making it autonomous: flapping between servers is disruptive and could lead to data inconsistency if you switch when replication is behind; data inconsistency is usually way worse than write downtime until someone logs in to flip the switch.
Try to have half your slaves off each master, so if a master is down, you still have 50% capacity. (I've seen some patches from google a while ago to keep binary logs in sync between masters, and make switching masters easy: If that's available, you may be able to have slaves just follow the current active master)
If you have budget for it, an extra slave off each master can be helpful: You can cron them to shutdown MySQL, tar up the directory, and restart. If you untar that on a new slave, it'll continue replicating from that point in time. If you rotate out the backups, you also have some ability to restore data from the past, if there is a bad update.
I don't get the hate towards linode here, on hacker news. I've been their client for a couple of years now and I find it an excellent vps provider. Excellent uptime and performance at a pretty good price. AWS has a few outages every year. Google just had one last week. Azure sucks balls. So, why the hate? Is it because it competes with some ycombinator startups?
Earlier quoted context omitted.
> it is not their fault I don't understand this line of reasoning. It's not like DDoS attacks are some kind of 0-day failure mode that nobody has seen before. Would you also say "it is not their fault" if their uplink provider had a fiber cut and they didn't have redundant uplinks? I'm guessing not: it's well understood that has a service provider you need to plan for this kind of unavailability and pay more money fo…
There are a handful of environments that can sustain a large, coordinated DDOS attack. Can you sink 10-20Gb/s of traffic forever? Not cost effectively.
That's for volumetric attacks (udp reflection), tls handshaking can eat all the CPU way before we run out of network :(
But now I think I need to setup fail-over with another VPS provider. What's a recommended alternative? Is Digital Ocean the next best choice after Linode?
Earlier quoted context omitted.
Apparently they came to the same conclusion—just a bit late. https://status.linode.com/incidents/frq9sz7dyb1x
What in that article makes you think that? I don't see it. They do say "we'll have to upgrade Xen nodes", but they don't mention the DDoS or link them.