Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mitigating a DDoS on Mastodon

coffee-and-dreams.uk

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Mitigating a DDoS on Mastodon

#31

Post by buro9 »

bloopernova wrote: post #29
buro9 wrote: post #23

Earlier quoted context omitted.

Cloudflare EM for DDoS Protection here. If a customer wants to hide their IP then the best way to do it: 1. Onboard onto Cloudflare 2. Audit your app and ensure you aren't leaking your IP (are you sending email directly? making web calls directly? - make adjustments to use APIs of other providers, i.e. send emails via Sendgrid API, etc) 3. Change your IP (it was previously public knowledge in your DNS records) At thi…

Is cloudflare affordable for an open source and low-funds project? (I honestly don't know the pricing, this isn't meant to be argumentative)

We have a free tier, and the caching and firewall is good enough on that tier - I use it :)

The DDoS protection is the same across all tiers - it's built in and you aren't charged for that. You even see other features (like the Rate Limit feature cited in the article) explicitly structure their pricing so that you are not charged for attack traffic even if you are on a paid plan or feature.

For small denial of service attacks the Security Level switch is very good at stopping the vast majority of attack traffic, and then the IP blocking and User Agent blocking is good too - this is available on the free plan, as are a handful of Firewall Rules that can allow complex expressions to match and drop traffic.

So you can get a very long way on the free plan.

Paid features I'd recommend if you want to stay on the free plan month-to-month yet go paranoid for a small cost:

1. Rate Limit, configure it on your dynamic endpoints to minimise the costs to you but have it highly effective against attacks. Predicted cost is relative to how many requests for dynamic endpoints you have... you can be smart here and combine with Firewall Rules to drop traffic that does not have auth credentials.

2. Argo Tunnel, to hide your IP.

There are other plan level benefits, and the most notable is the quantity of Firewall Rules per plan level and the complexity they allow: https://www.cloudflare.com/en-gb/plans/

Re: Mitigating a DDoS on Mastodon

#32

Post by koroshitekure »

buro9 wrote: post #23
ekimekim wrote: post #4

On the subject of the IP leaking: Note that IPv4 only has 2^32 addresses, and people can and do mass scan all of them (see here shodan.io). If your service is exposing any identifiable information (ie. if it's not completely blocking all non-cloudflare IPs) then it's fairly easy to find even if it's "unguessable".

Cloudflare EM for DDoS Protection here. If a customer wants to hide their IP then the best way to do it: 1. Onboard onto Cloudflare 2. Audit your app and ensure you aren't leaking your IP (are you sending email directly? making web calls directly? - make adjustments to use APIs of other providers, i.e. send emails via Sendgrid API, etc) 3. Change your IP (it was previously public knowledge in your DNS records) At thi…

hey, OP here

I'm no longer on keybase, deleted it a few days ago - but I'm more than happy to share what I found if you want

pretty sure it's nothing groundbreaking though

other contact methods are listed on my profile

(edit: by OP I mean article author)

Re: Mitigating a DDoS on Mastodon

#33

Post by koroshitekure »

dredmorbius wrote: post #24
koroshitekure wrote: post #11

hey, i'm the author of the article really... surprised it got submitted here incidentally i'm running pleroma, not mastodon. minor detail but you know

I'm quite active on Mastodon and HN, thought this might be of interest. Would you prefer the title were modified? The mods can do that. I thought that specifying what the DDoS mitigation was applied to would be helpful, though my presumption of Mastodon was in error, apologies.

I'm not too bothered about correcting it, just thought it good to note

Re: Mitigating a DDoS on Mastodon

#34

Post by chvanikoff »

koroshitekure wrote: post #16
iampims wrote: post #15

Earlier quoted context omitted.

To avoid leaking IPs, you can use cloudflared tunnel. It might get pricy if you move a lot of bytes, but it’ll isolate you from IP leaking issues.

oh, i found out where the leak was it's right at the end of the article - the attacker was abusing the "create a preview card of any posted URL" feature - he'd post a link, wait for pleroma to go and grab the url to preview it, then narrow down which one was mine based on user agent i added an upstream proxy and anonymised the user agent, so even if he were to do that, the most he'd find was my proxy box

That might be what you are talking about, but just to confirm: Pleroma has an ability to proxy outbound requests via `pleroma.http` config out of the box

Re: Mitigating a DDoS on Mastodon

#35

Post by koroshitekure »

chvanikoff wrote: post #34
koroshitekure wrote: post #16

Earlier quoted context omitted.

oh, i found out where the leak was it's right at the end of the article - the attacker was abusing the "create a preview card of any posted URL" feature - he'd post a link, wait for pleroma to go and grab the url to preview it, then narrow down which one was mine based on user agent i added an upstream proxy and anonymised the user agent, so even if he were to do that, the most he'd find was my proxy box

That might be what you are talking about, but just to confirm: Pleroma has an ability to proxy outbound requests via `pleroma.http` config out of the box

yeah that's what I'm using

I also pull-requested a user agent anonymisation setting (pleroma.http.user_agent) to make this better

Re: Mitigating a DDoS on Mastodon

#36

Post by bloopernova »

buro9 wrote: post #31
bloopernova wrote: post #29

Earlier quoted context omitted.

Is cloudflare affordable for an open source and low-funds project? (I honestly don't know the pricing, this isn't meant to be argumentative)

We have a free tier, and the caching and firewall is good enough on that tier - I use it :) The DDoS protection is the same across all tiers - it's built in and you aren't charged for that. You even see other features (like the Rate Limit feature cited in the article) explicitly structure their pricing so that you are not charged for attack traffic even if you are on a paid plan or feature. For small denial of servic…

Thank you, that's a lot of detail and I appreciate you taking the time to respond.

Re: Mitigating a DDoS on Mastodon

#37

Post by bsysop »

zaarn wrote: post #17
ekimekim wrote: post #4

On the subject of the IP leaking: Note that IPv4 only has 2^32 addresses, and people can and do mass scan all of them (see here shodan.io). If your service is exposing any identifiable information (ie. if it's not completely blocking all non-cloudflare IPs) then it's fairly easy to find even if it's "unguessable".

Well, that would only work if the other end responds to a request to the IP address with a cert that includes the proper domain. If you setup Cloudflare properly, then you only see a CF-based certificate, not that actual hostnames. Since you didn't send a proper hostname (unless you use PTR, which isn't reliable either) it'll use whatever default hostname it has configured (or just close the connection). Or in a case…

This is huge. There are a ton of mis-configured Apache and nginx reverse proxies out there that expose the primary domain name of the site being served. You can quickly test this for yourself by running "curl -vk https://your.ip.address" and see what pops up for the CN field or Location header.

Even worse is the pattern of requesting LetsEncrypt certificates for multiple domains on one certificate. Now all of a sudden you're leaking development server hostnames, peeling off the white label of multi-tenant, and making things easier for automated scanners.

I get it that security by hostname obscurity is a poor practice on its own, but there's also something to be said for cutting down a large amount of malicious traffic with some common best practices.

Re: Mitigating a DDoS on Mastodon

#38

Post by zaarn »

bsysop wrote: post #37
zaarn wrote: post #17

Earlier quoted context omitted.

Well, that would only work if the other end responds to a request to the IP address with a cert that includes the proper domain. If you setup Cloudflare properly, then you only see a CF-based certificate, not that actual hostnames. Since you didn't send a proper hostname (unless you use PTR, which isn't reliable either) it'll use whatever default hostname it has configured (or just close the connection). Or in a case…

This is huge. There are a ton of mis-configured Apache and nginx reverse proxies out there that expose the primary domain name of the site being served. You can quickly test this for yourself by running "curl -vk https://your.ip.address" and see what pops up for the CN field or Location header. Even worse is the pattern of requesting LetsEncrypt certificates for multiple domains on one certificate. Now all of a sudde…

Hence I use Wildcard LE certs, it helps a lot as well as using bogus or non-CA'd certificates if no host name is supplied (or just sending 0 byte pages with no useful data)

Re: Mitigating a DDoS on Mastodon

#39

Post by pjc50 »

Decentralisation fans take note: despite wanting to remain independent, the only effective solution was in this case to re-insert a giant global intermediary (Cloudflare) and block all the anonymous unaccountable Tor users.

If a decentralised system is to stay decentralised, it needs to consider spammy bad actors.

Re: Mitigating a DDoS on Mastodon

#40

Post by lifty »

pjc50 wrote: post #39

Decentralisation fans take note: despite wanting to remain independent, the only effective solution was in this case to re-insert a giant global intermediary (Cloudflare) and block all the anonymous unaccountable Tor users. If a decentralised system is to stay decentralised, it needs to consider spammy bad actors.

That's true, Cloudflare has mastered the art of DDoS mitigation and they have developed some amazing tools [1] to achieve that, and fortunately they are sharing some of this knowledge. With the advent of eBPF, I reckon that this kind of tooling will become more accessible and easy to deploy for people that do self-hosting. I also hope that DDoS mitigations based on web of trust or other type of cryptographic identity [2] will come about in the future, although I wouldn't hold my breath for that.

[1] https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig... [2] https://identity.foundation

Post reply on HN