Is that me, or did CloudFlare outages increase since LLM "engineers" were hired remotely? Do you think there is a correlation?
They've always been flakey. At least these only impacted their own customers instead of taking down the internet.
Cloudflare outage on December 5, 2025
241–250 of 601 posts
Re: Cloudflare outage on December 5, 2025
#242Earlier quoted context omitted.
[flagged]
Indeed, but it is what it is. Cloudflare comes out of my budget, and even with downtime, its better than not paying them. Do I want to deal with what Cloudflare offers? I do not, I have higher value work to focus on. I want to pay someone else to deal with this, and just like when cloud providers are down, it'll be back up eventually. Grab a coffee or beer and hang; we aren't savings lives, we're just building websit…
> Do I want to deal with what Cloudflare offers? I do not, I have higher value work to focus on. I want to pay someone else to deal with this, and just like when cloud providers are down, it'll be back up eventually.
This is specious reasoning. How come I had to endure a total outage due to the rollout of a mitigation of a Nextjs vulnerability when my organization doesn't even own any React app, let alone a Nextjs one?
Also specious reasoning #2, not wanting to maintain a service does not justify blindly rolling out config changes globally without any safeguards.
Re: Cloudflare outage on December 5, 2025
#243[dead]
Re: Cloudflare outage on December 5, 2025
#244Messing around on a Friday? Brave.
Or overworked.
We can deploy on Fridays. We don't, because we aren't donating our time to the shareholders.
Re: Cloudflare outage on December 5, 2025
#245Earlier quoted context omitted.
That bit may be true, but the underlying error of a null reference that caused a panic was exactly the same in both incidents.
Yep: it is wild for them to claim that a strongly-typed language would have saved them when it didn't. Relying on language features instead of writing code well will always eventually backfire.
You're right that you have to "write code well" to prevent this sort of thing. It's also true that Rust's language features, if you use them, can make this sort of mistake a compile-time error rather than something that only blows up at runtime under the wrong conditions. The problem with their last outage was that somebody explicitly opted out of the tool provided by the language. As you say, that's "not writing code well". But I think you're dismissing the value of the language feature in helping you write code well.
Re: Cloudflare outage on December 5, 2025
#246A lot of these kind of bugs feel like they could be caught be a simple review bot like Greptile... I wonder if Cloudlare uses an equivalent tool internally?
That has not been my experience with those tools.
Super-procedural code in particular is too complex for humans to follow, much less AI.
Re: Cloudflare outage on December 5, 2025
#247Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.
Indeed. AWS too.
I feel like the cloud hosting companies have lost the plot. "They can provide better uptime than us" is the entire rationale that a lot of small companies have when choosing to run everything in the cloud.
If they cost more AND they're less reliable, what exactly is the reason to not self host?
Re: Cloudflare outage on December 5, 2025
#248Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…
"Kudos"? This is like the South Park episode in which the oil company guy just excuses himself while the company just continues to fuck up over and over again. There's nothing to praise, this shouldn't happen twice in a month. Its inexcusable.
twice in a month _so far_
Re: Cloudflare outage on December 5, 2025
#249Earlier quoted context omitted.
This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.
... until the extended-range version is ordered and no one remembers to fix the leak. :]
They will remember, because it'll have been measured and documented, rigorously.
Re: Cloudflare outage on December 5, 2025
#250Earlier quoted context omitted.
> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…
Genuinely curious, how to actually implement detection systems for a large scale global infra which that works with < 1 minute SLO ? Given cost is no constraint.
Right now I'd say maybe don't push changes to your entire global infra all at once and certainty not without testing your change first to make sure it doesn't break anything, but it's really not about a specific failure/fix as much as it is about a single company getting too big to do the job well or just plain doing more than it should in the first place.
Honestly we shouldn't have created a system where any single company's failure is able to impact such a huge percentage of the network. The internet was designed for resilience and we abandoned that ideal to put our trust in a single company that maybe isn't up for the job. Maybe no one company ever could do it well enough, but I suspect that no single company should carry that responsibility in the first place.