[deleted]
Cloudflare Global Network experiencing issues
971–980 of 1001 posts
Re: Cloudflare Global Network experiencing issues
#972How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.
All decentralized systems tend to centralization over time.
Re: Cloudflare Global Network experiencing issues
#973> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
Configuration changes are dangerous for CF it seems, and knocked down $NET almost 4% today. I wonder what the industry wide impact is for each of these outages?
Pre market was red for all tech stocks today before the outage even happened
Re: Cloudflare Global Network experiencing issues
#974> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom. You NEED to phase config rollouts like you phase code rollouts.
The big dogs absolutely do phase config rollouts as a general rule.
There are still two weaknesses:
1) Some configs are inherently global and cannot be phased. There's only one place to set them. E.g. if you run a webapp, this would be configs for the load balancer as opposed to configs for each webserver
2) Some configs have a cascading effect -- even though a config is applied to 1% of servers, it affects the other servers they interact with, and a bad thing spreads across the entire network
Re: Cloudflare Global Network experiencing issues
#975Has anyone else noticed a major drop in email spam with this cloudflare outage?
Re: Cloudflare Global Network experiencing issues
#976one way to mitigate DDoS is to enforce source IP checks on the way OUT of a datacenter (egress).
sure there are botnets, infected devices, etc that would conform to this but where does the sheer power of a big ddos attack come from? including those who sell it as a service. they have to have some infrastructure in some datacenter right?
make a law that forces every edge router of a datacenter to check for source IP and you would eliminate a very big portion of DDoS as we know it.
until then, the only real and effective method of mitigating a DDoS attack is with even more bandwidth. you are basically a black hole to the attack, which cloudflare basically is.
Re: Cloudflare Global Network experiencing issues
#977one way to mitigate DDoS is to enforce source IP checks on the way OUT of a datacenter (egress). sure there are botnets, infected devices, etc that would conform to this but where does the sheer power of a big ddos attack come from? including those who sell it as a service. they have to have some infrastructure in some datacenter right? make a law that forces every edge router of a datacenter to check for source IP a…
The biggest attacks literally come from botnets. There’s not a lot coming from infrastructure services precisely because these services are incentivized to shut that shit down. At most it would be used as the control plane which is how people attempt to shut down the botnets.
Re: Cloudflare Global Network experiencing issues
#978If anyone needs commands for turning off the CF proxy for their domains and happens to have a Cloudflare API token. First you can grab the zone ID via: curl -X GET "https://api.cloudflare.com/client/v4/zones" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[] | "\(.id) \(.name)"' And a list of DNS records using: curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID…
I would advise against this action. Just ride the crash.
Re: Cloudflare Global Network experiencing issues
#979This incident has been resolved.
Posted 4 minutes ago. Nov 18, 2025 - 19:28 UTC
Better link for chroniclers, since the incident is now buried pretty far down on the status page: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
Re: Cloudflare Global Network experiencing issues
#980This incident has been resolved. Posted 4 minutes ago. Nov 18, 2025 - 19:28 UTC Better link for chroniclers, since the incident is now buried pretty far down on the status page: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
Thanks! We've switched to that from https://www.cloudflarestatus.com/?t=1 at the top.