Time for Cloudflare to start using the BOFH excuse generator. https://bofh.d00t.org/
[deleted]
Cloudflare outage on December 5, 2025
221–230 of 601 posts
Re: Cloudflare outage on December 5, 2025
#222Earlier quoted context omitted.
Cloudflare is orders of magnitude larger than any fintech. Rollouts likely take much longer, and having a human monitoring a dashboard doesn't scale.
That means they engineered their systems incorrectly then? Precisely because they are much bigger, they should be more resilient. You know who's bigger than Cloudflare? tier-1 ISPs, if they had an outage the whole internet would know about it, and they do have outages except they don't cascade into a global mess like this. Just speculating based on my experience: It's more likely than not that they likely refused to…
> Control-plane and data-plane should be separate
They are separate.
> a react patch shouldn't affect traffic forwarding.
If you can’t even bother to read the blog post maybe you shouldn’t be so confident in your own analysis of what should and shouldn’t have happened?
This was a configuration change to change the buffered size of a body from 256kb to 1mib.
The ability to be so wrong in so few words with such confidence is impressive but you may want to take more of a curiosity first approach rather than reaction first.
Re: Cloudflare outage on December 5, 2025
#223Earlier quoted context omitted.
In fairness, the previous bug (with the Rust unwrap ) should never have happened: someone explicitly called the panicking function, the review didn't catch it and the CI didn't catch it. It required a significant organizational failure to happen. These happen but they ought to be rarer than your average bug (unless your organization is fundamentally malfunctioning, that is)
The issue would also not have happened, if someone did the right code, tests, and the review or CI caught it...
It's different to expect somebody to write the correct program every time than to expect somebody not to call the "break_my_system" procedure that was warnings all over it telling people it's there for quick learning-to-use examples or other things you'll never run.
Re: Cloudflare outage on December 5, 2025
#224Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.
TBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic. I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy…
Costco had a deal on solid-state UPS & solar panels a while back that I was happy to partake of
Re: Cloudflare outage on December 5, 2025
#225Earlier quoted context omitted.
> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…
Can you name a major cloud provider that doesn’t have major outages? If this were purely a money problem it would have been solved ages ago. It’s a difficult problem to solve. Also, they’re the youngest of the major cloud providers and have a fraction of the resources that Google, Amazon, and Microsoft have.
> Can you name a major cloud provider that doesn’t have major outages?
That fact that no major cloud provider is actually good is not an argument that cloudflare isn't bad, or even that they couldn't/shouldn't do better than they are. They have fewer resources than Google or Microsoft but they're also in a unique position that makes us differently vulnerable when they fuck up. It's not all their fault, since it was a mistake to centralize the internet to the extent that we have in the first place, but now that they are responsible for so much they have to expect that people will be upset when they fail.
Re: Cloudflare outage on December 5, 2025
#226Dang… I don’t even use React and it still brings down my sites. Good beats I guess.
Re: Cloudflare outage on December 5, 2025
#227My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once". But a more important takeaway: > This type of code error is prevented by languages with strong type systems
This is the exact same type of error that happened in their Rust code last time. Strong type systems don’t protect you from lazy programming.
It's not remotely the same type of error -- error non-handling is very visible in the Rust code, while the Lua code shows the happy path, with no indication that it could explode at runtime.
Perhaps it's the similar way of not testing the possible error path, which is an organizational problem.
Re: Cloudflare outage on December 5, 2025
#228What's the culture like at Cloudflare re: ops/deployment safety? They saw errors related to a deployment, and because it was related to a security issue instead of rolling it back they decided to make another deployment with global blast radius instead? Not only did they fail to apply the deployment safety 101 lesson of "when in doubt, roll back" but they also failed to assess the risk related to the same deployment…
> this sounds like the sort of cowboy decision Ouch. Harsh given that Cloudflare's being over-honest (to disabling the internal tool) and the outage's relatively limited impact (time wise & no. of customers wise). It was just an unfortunate latent bug: Nov 18 was Rust's Unwrap, Dec 5 its Lua's turn with its dynamic typing. Now, the real cowboy decision I want to see is Cloudflare [0] running a company-wide Rust/Lua c…
[deleted]
Re: Cloudflare outage on December 5, 2025
#229Earlier quoted context omitted.
That is also true at Cloudflare for what it’s worth. However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release, especially since there’s a 5 min lag (if I recall correctly) in the monitoring dashboards to get all the telemetry from thousands of servers worldwide. Comparing the difficulty of running the world’s internet traffic w…
> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…
Genuinely curious, how to actually implement detection systems for a large scale global infra which that works with < 1 minute SLO ? Given cost is no constraint.
Re: Cloudflare outage on December 5, 2025
#230I notice that this is the kind of thing that solid sociable tests ought to have caught. I am very curious how testable that code is (random procedural if-statements don't inspire high confidence.)