Oh good, a new way to break things; sure hope nobody using this expands into new markets or gets a surge of traffic outside of their usual viewers!
That's an awful lot and weird choice of words just to say that you are not the target audience.
Cloudflare Adaptive DDoS Protection
11–20 of 27 posts
Re: Cloudflare Adaptive DDoS Protection
#12This is very similar to Google’s Cloud Armour Adaptive Protection[1] which recently fended off a 46 million RPS DDoS attack[2]. [1] https://cloud.google.com/armor/docs/adaptive-protection-over... [2] https://cloud.google.com/blog/products/identity-security/how...
> Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes.
Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be bad perhaps?
This is exactly, why a more appropriate mitigation is to issue connection-close (in http1.1 world) or GOAWAY frame (in http2 world).
With that in hand, it's possible to bounce the connection back to the bot. Sure, the bot will retry which will put pressure on TLS. But then, if a source IP is trying very hard to establish many connections it's another strong sign of malicious traffic. In TCP you can rate limit SYN packets and _not_ allow more than X new connections per second, quite easily. Another mitigation is to allow the bot to send http requests, but just not answer them. Allow them hang on the connection, burning the bot's resources (memory, concurrent connetions).
Anyway, the point is: requests-per-second is not a perfect metric for L7 / application attacks. A smart application shoudl be albe to push mitigation up to lower levels (like syn rate limiting, or TLS client-hello signature rate limiting), and _not_ intake large amounts of traffic. A better metric is bot count (or number of unique IP's seen), which was shared in the article and is in mid-range.
To recap, a well designed system should _never_ need to handle 47MRPS. It should be able to packpressure it.
Re: Cloudflare Adaptive DDoS Protection
#13FYI - this is Cloudflare "GA week" [0]. So we're going to see a bunch of beta/early-adopter service made GA this week.
Other services made GA today, at the time of me writing this post:
- Cloudflare One [1]
- Cloudflare WAF [2]
- Domain Scoped Roles [3]
[0] https://blog.cloudflare.com/welcome-to-ga-week/
[1] https://blog.cloudflare.com/cloudforce-one-is-now-ga/
Re: Cloudflare Adaptive DDoS Protection
#14Earlier quoted context omitted.
Sigh. You don't have to use this. Large companies that want super tailored DDoS protection can. Everyone else can just use our standard DDoS protection which is included with every plan and works: https://blog.cloudflare.com/26m-rps-ddos/
Isn't it frustrating when users complain about new features that they aren't expected to use? Why is it like that? Is it like this in other industries? Do people that use Coach bags complain when Coach introduces a wallet that they won't use? I simply do not understand it.
Not commenting on this particular case, but I know in smaller dev shops that time spent working on one feature is directly taken away from time spent on other things, and so if the new feature isn't useful to you, it has also taken fixing and dev time away from ones that might be. I also suspect that many people have seen the trajectory where apps get more and more features and don't focus on their core competency.
Again, I don't believe this is the case here, but it might generally explain the phenomena/misplaced snark.
Re: Cloudflare Adaptive DDoS Protection
#15I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules, but for their network vs an appliance.
Re: Cloudflare Adaptive DDoS Protection
#16I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules,…
Did you work with someone at Cloudflare on this?
Re: Cloudflare Adaptive DDoS Protection
#17I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules,…
Did you work with someone at Cloudflare on this?
Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up so kudos to your team. This intermittent random UDP issue is more recent while I was trying the second host, but couldn't notice a pattern or replicate, so I ended up rolling my own with tc, port knocking, and the largest pipe we could find.
Edit: For others, our problem was probably an edge case, and our use is probably not common. MT seems good compared to the alternative solutions I know so looking forward for an opp to try this if I have ever a need.
Re: Cloudflare Adaptive DDoS Protection
#18Earlier quoted context omitted.
Did you work with someone at Cloudflare on this?
Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up…
Pity. I love debugging weird network things.
Re: Cloudflare Adaptive DDoS Protection
#19This is very similar to Google’s Cloud Armour Adaptive Protection[1] which recently fended off a 46 million RPS DDoS attack[2]. [1] https://cloud.google.com/armor/docs/adaptive-protection-over... [2] https://cloud.google.com/blog/products/identity-security/how...
> Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes. Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be…
Google doesn't terminate TLS on "the servers". It gets terminated on the frontend load balancers (GFE), which is presumably where the attack mitigation happened.
Re: Cloudflare Adaptive DDoS Protection
#20Earlier quoted context omitted.
I am running quite a different browser setup from most regular web users. I am using Firefox with multi-account containers and uBlock Origin enabled, and I also have an OpenVPN client running, the amount of captchas and distrust I receive from Cloudflare (and their customers, unknowingly I suppose) feels disproportionate. It's funny to me that these companies most likely spends a lot of time and energy on how to opti…
CAPTCHAs are going away: https://blog.cloudflare.com/end-cloudflare-captcha/
I see a lot of sites (including those behind Cloudflare) being blocked when using Cloudfare's Warp VPN. 403s, captchas, etc.
It would be nice if Cloudflare could put abuse control measures on the front-side, and stop the abuse before allowing it to exit their VPN. E.g., if Cloudflare detects abuse with their existing anti-DDoS, anti-scraping, etc. measures, and the client is on Warp, instead of creating a poor experience for every other Warp user sharing that exit IP, just block/aggressively throttle the bad actor's traffic at the VPN terminator.