> A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.
Cloudflare was down
421–430 of 560 posts
Re: Cloudflare was down
#422Earlier quoted context omitted.
Agreed, I feel bad for them. But mostly because cloudflare's workflows are so bad that you're seemingly repeatedly set up for really public failures. Like how does this keep happening without leadership's heads rolling. The culture clearly is not fit for their level of criticality
> The culture clearly is not fit for their level of criticality I don't think anyone's is.
How often do you hear of Akamai going down and they host a LOT more enterprise/high value sites than Cloudflare.
There's a reason Cloudflare has been really struggling to get into the traditional enterprise space and it isn't price.
Re: Cloudflare was down
#423This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.
We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is. It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime. Therefore, many engineers at affected companies would ha…
On the other thread there were comments claiming it’s unknowable what IaaS some SaaS is using, but SaaS vendors need to disclose these things one way or another, e.g. DPAs. Here is for example renders list of subprocessors: https://render.com/security
It’s actually fairly easy to know which 3rd party services a SaaS depends on and map these risks. It’s normal due diligence for most companies to do so before contracting a SaaS.
Re: Cloudflare was down
#424> A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.
Where’s the source for this?
It doesn’t look good when similar WAF issues caused their big outage a few years back.
Re: Cloudflare was down
#425Earlier quoted context omitted.
> Why didn't several other such popular startups appear bunny.net fastly.com gcore.com keycdn.com Cloudfront Probably some more I forgot now. CF is not the only option and definitely not the best option. > Yeah, now we'll save everyone from DDoS, everything's perfect, we'll speed up your site, ... and host the providers selling DDoS services. https://privacy-pc.com/articles/spy-jacking-the-booters.html
Thank you for sending these alternatives, they look good. And, of course, the most important thing is that Cloudflare is free, while these alternatives cost money. And they cost hundreds of dollars at my traffic volume of tens of terabytes. Of course, I really don't want to pay. So, as they say, mice wept and jabbed, but they kept gnawing on the cactus.
Nothing's free - one day they will come knocking. Better be prepared to serve at an affordable level.
Re: Cloudflare was down
#426Earlier quoted context omitted.
This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.
But the nature of a CDN and most other products CF offers, is central by nature. If you switch from CF to the next CF competitor, you've not improved this dependency. The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewher…
yeah there is no incentive to do a CDN in house, esp for businesses that are not tech-oriented. And the costs of the occasional outage has not really been higher than the cost of doing it in-house. And I'm sure other CDNs gets outages as well, just CF is so huge everyone gets to know about it and it makes the news
Re: Cloudflare was down
#427Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them. Wise was just down which is a pretty big one. Also odd how some websites were down…
The excuse: >A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. >The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. >We will share more information as we have it today. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q
It's quite an unfortunate coincidence that React has indirectly been the reason for two recent issues at Cloudflare haha
Re: Cloudflare was down
#428Earlier quoted context omitted.
But it might have something to do with the "rewrite" part: > The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive. > Back to that two page function. Yes, I know, it’s just a simple function to display a window, but i…
A lot of words for a 'might'. We don't know what caused the downtime.
Not this time; but the rewrite was certainly implicated in the previous one. They actually had two versions deployed; in response to unexpected configuration file size, the old version degraded gracefully, while the new version failed catastrophically.
Re: Cloudflare was down
#429Earlier quoted context omitted.
But the nature of a CDN and most other products CF offers, is central by nature. If you switch from CF to the next CF competitor, you've not improved this dependency. The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewher…
You can load-balance between CDN vendors as well
With what? The only (sensible) way is DNS, but then your DNS provider is your SPOF. Amazon used to run 2 DNS providers (separate NS from 2 vendors for all of AWS), but when one failed, there was still a massive outage.
Re: Cloudflare was down
#430That's the 30% vibe code they promised us. Cynicism aside, something seems to be going wrong in our industry.
> Cynicism aside, something seems to be going wrong in our industry.
Started after the GFC and the mass centralisation of infrastructure