Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mitigating a DDoS on Mastodon

coffee-and-dreams.uk

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Mitigating a DDoS on Mastodon

#61

Post by zzzcpan »

voidwtf wrote: post #58
zzzcpan wrote: post #57

Earlier quoted context omitted.

It was far from the only effective solution. Probably just an easy path for someone who has no idea about DDoS attacks, but influenced by advertisement and propaganda. Volumetric attacks don't actually require centralized global intermediaries to mitigate, there are other ways to do it, and Layer 7 attacks are even application specific and should be handled by applications or by someone running them who understands a…

Can you provide an example of how to mitigate a volumetric attack without significant reliance on intermediaries?

Say you have a few nodes behind a few different ISPs sharded to clients. Once one node becomes unavailable it gets replaced by another node and back when it becomes available again. This means either all nodes at once can get attacked, but with lower volume or one by one, but affecting only one shard of users for a short period of time it takes to failover.

But in practice datacenters, uplinks and internet exchanges often are able to do flowspec, firewall rules, block all UDP for a subnet in all networks they have relationships with, etc. So plenty of those nodes can be behind ISPs that mitigate volumetric attacks automatically, so even simple DNS failover might be good enough to protect from such attacks. It's not that hard. Layer 7 is where the hard part is.

Re: Mitigating a DDoS on Mastodon

#62

Post by voidwtf »

zzzcpan wrote: post #61
voidwtf wrote: post #58

Earlier quoted context omitted.

Can you provide an example of how to mitigate a volumetric attack without significant reliance on intermediaries?

Say you have a few nodes behind a few different ISPs sharded to clients. Once one node becomes unavailable it gets replaced by another node and back when it becomes available again. This means either all nodes at once can get attacked, but with lower volume or one by one, but affecting only one shard of users for a short period of time it takes to failover. But in practice datacenters, uplinks and internet exchanges…

For reference, having been in this situation before:

1. Having had ~25 servers per datacenter

2. 5 Datacenters (1 in Texas, 1 in Utah, 2 in California, 1 in Chicago)

3. 1 Server in each location connected 10gbit, the rest 1 gbit.

I got to watch first hand as DNS reflection attacks crippled our infrastructure one server at a time. Only 2 of the datacenters (1 in LA, 1 in Chicago) had the infrastructure to mitigate the DDoS without significantly effecting their operations. Even post mitigation, the 2 datacenters that didn't end-up blackhole-ing our IPs at the edge still let so much malicious traffic through that only the 2 10Gbit servers remained online and they were nearly CPU bound over 24 cores just handling all the SENDQ/RECVQ for the NIC.

I mention this because it's sometimes easy to dismiss until you're in the situation and the realities of what you have control over are vastly different from technically feasible. The size and scope of modern DDoS attacks can easily overwhelm entire uplinks to a datacenter, even after pushing mitigations upstream. The reason these reverse proxies from companies like Cloudflare have become so popular is because most will not have the raw resources required to mitigate this themselves. Even some larger datacenters don't have the resources.

Re: Mitigating a DDoS on Mastodon

#63

Post by adventskalender »

Ever since they injected 70kb of JavaScript into my site, I don't trust Cloudflare anymore. I just checked, and other CDNs seem to be very expensive.

I wonder if there are any other ways to defend against DDoS?

Maybe looking for a host that helps with such matters? But then they will probably be more expensive, too?

Re: Mitigating a DDoS on Mastodon

#64

Post by zzzcpan »

voidwtf wrote: post #62
zzzcpan wrote: post #61

Earlier quoted context omitted.

Say you have a few nodes behind a few different ISPs sharded to clients. Once one node becomes unavailable it gets replaced by another node and back when it becomes available again. This means either all nodes at once can get attacked, but with lower volume or one by one, but affecting only one shard of users for a short period of time it takes to failover. But in practice datacenters, uplinks and internet exchanges…

For reference, having been in this situation before: 1. Having had ~25 servers per datacenter 2. 5 Datacenters (1 in Texas, 1 in Utah, 2 in California, 1 in Chicago) 3. 1 Server in each location connected 10gbit, the rest 1 gbit. I got to watch first hand as DNS reflection attacks crippled our infrastructure one server at a time. Only 2 of the datacenters (1 in LA, 1 in Chicago) had the infrastructure to mitigate the…

> I mention this because it's sometimes easy to dismiss until you're in the situation and the realities of what you have control over are vastly different from technically feasible.

I understand, but you are still talking about a situation where surviving a volumetric DDoS attack without a global centralized provider was possible. It wasn't smooth for you, but it could have been if things were done a bit differently.

Anyway, here on the other side of the world it's not like that, DDoS protection is more common. Because in the early days of DDoS attacks with all the dreadful blackholing one of the big European providers OVH invested in DDoS protection and kind of pushed the whole market to provide it too instead of blackholing.

Re: Mitigating a DDoS on Mastodon

#65

Post by voidwtf »

zzzcpan wrote: post #64
voidwtf wrote: post #62

Earlier quoted context omitted.

For reference, having been in this situation before: 1. Having had ~25 servers per datacenter 2. 5 Datacenters (1 in Texas, 1 in Utah, 2 in California, 1 in Chicago) 3. 1 Server in each location connected 10gbit, the rest 1 gbit. I got to watch first hand as DNS reflection attacks crippled our infrastructure one server at a time. Only 2 of the datacenters (1 in LA, 1 in Chicago) had the infrastructure to mitigate the…

> I mention this because it's sometimes easy to dismiss until you're in the situation and the realities of what you have control over are vastly different from technically feasible. I understand, but you are still talking about a situation where surviving a volumetric DDoS attack without a global centralized provider was possible. It wasn't smooth for you, but it could have been if things were done a bit differently.…

This is true, it's certainly gotten better since OVH introduced it. It's not so niche as BlackLotus, GigeNET, Arbor Networks, etc...

Re: Mitigating a DDoS on Mastodon

#66

Post by jabot »

koroshitekure wrote: post #32
buro9 wrote: post #23

Earlier quoted context omitted.

Cloudflare EM for DDoS Protection here. If a customer wants to hide their IP then the best way to do it: 1. Onboard onto Cloudflare 2. Audit your app and ensure you aren't leaking your IP (are you sending email directly? making web calls directly? - make adjustments to use APIs of other providers, i.e. send emails via Sendgrid API, etc) 3. Change your IP (it was previously public knowledge in your DNS records) At thi…

hey, OP here I'm no longer on keybase, deleted it a few days ago - but I'm more than happy to share what I found if you want pretty sure it's nothing groundbreaking though other contact methods are listed on my profile (edit: by OP I mean article author)

On an unrelated note: why are you no longer on keybase? Are there any problems with the service?

Re: Mitigating a DDoS on Mastodon

#67

Post by koroshitekure »

jabot wrote: post #66
koroshitekure wrote: post #32

Earlier quoted context omitted.

hey, OP here I'm no longer on keybase, deleted it a few days ago - but I'm more than happy to share what I found if you want pretty sure it's nothing groundbreaking though other contact methods are listed on my profile (edit: by OP I mean article author)

On an unrelated note: why are you no longer on keybase? Are there any problems with the service?

the guy that was attacking me attempted to dox me, so I shut down all non-essential accounts to prevent anything similar happening in the future
Post reply on HN