Time to consider alternatives: https://x.com/GithubProjects/status/1990804801811329329
Cloudflare Global Network experiencing issues
991–1000 of 1001 posts
Re: Cloudflare Global Network experiencing issues
#992> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
Configuration changes are dangerous for CF it seems, and knocked down $NET almost 4% today. I wonder what the industry wide impact is for each of these outages?
>Configuration changes are dangerous for CF it seems, and knocked down $NET almost 4% today. I wonder what the industry wide impact is for each of these outages?
This is becoming the "new normal." It seems like every few months, there's another "outage" that takes down vast swathes of internet properties, since they're all dependent on a few platforms and those platforms are, clearly, poorly run.
This isn't rocket surgery here. Strong change management, QA processes and active business continuity planning/infrastructure would likely have caught this (or not), as is clear from other large platforms that we don't even think about because outages are so rare.
Like airline reservations systems[0], credit card authorization systems from VISA/MasterCard, American Express, etc.
Those systems (and others) have outages in the "once a decade" or even much, much, longer ranges. Are the folks over at SABRE and American Express that much smarter and better than Cloudflare/AWS/Google Cloud/etc.? No. Not even close. What they are is careful as they know their business is dependent on making sure their customers can use their services anytime/anywhere, without issue.
It amazes me the level of "Stockholm Syndrome"[1] expressed by many posting to this thread, expressing relief that it wasn't "an attack" and essentially blaming themselves for not having the right tools (API keys, etc.) to recover from the gross incompetence of, this time at least, Cloudflare.
I don't doubt that I'll get lots of push back from folks claiming, "it's hard to do things at scale," and/or "there are way too many moving parts," and the like.
Other organizations like the ones I mention above don't screw they're customers every 4-6 months with (clearly) insufficiently tested configuration and infrastructure changes.
Yet many here seem to think that's fine, even though such outages are often crushing to their businesses. But if the customers of these huge providers don't demand better, they'll only get worse. And that's not (at least in my experience) a very deep or profound idea.
[0] https://en.wikipedia.org/wiki/Airline_reservations_system
Re: Cloudflare Global Network experiencing issues
#993I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.
Re: Cloudflare Global Network experiencing issues
#994deny (clippy:: unwrap_used)
Re: Cloudflare Global Network experiencing issues
#995[deleted]
Re: Cloudflare Global Network experiencing issues
#996is this the result of AI software improvements?
Re: Cloudflare Global Network experiencing issues
#997If anyone needs commands for turning off the CF proxy for their domains and happens to have a Cloudflare API token. First you can grab the zone ID via: curl -X GET "https://api.cloudflare.com/client/v4/zones" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[] | "\(.id) \(.name)"' And a list of DNS records using: curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID…
[dead]
Re: Cloudflare Global Network experiencing issues
#998If anyone needs commands for turning off the CF proxy for their domains and happens to have a Cloudflare API token. First you can grab the zone ID via: curl -X GET "https://api.cloudflare.com/client/v4/zones" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[] | "\(.id) \(.name)"' And a list of DNS records using: curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID…
thanks for this! just expanded on a bit and published a write up here so it's easier to find in the future: https://www.coryzue.com/writing/cloudflare-dns/
Re: Cloudflare Global Network experiencing issues
#999Is it me, or do the outages of single points of failure for large swaths of the internet tend to cluster within weeks/days of one another?
Anyone know why? Could be totally bias because one news story propels the next, so when they happen in clusters, you just hear about them more than when they don't.
Re: Cloudflare Global Network experiencing issues
#1000Cloudflare issue is due to latency in DDOS errors.
>cups.servic
>foomaticrip
Form a [cerulean] type-font in the page-source.