Cloudflare outage on December 5, 2025 • Hacker News Board

Re: Cloudflare outage on December 5, 2025

#341

Post by TechniKris » Fri, Dec 05, 2025, 10:04 PM UTC

cnnlives265 wrote: post #315

jacobgkau wrote: post #288

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

This is the case with just about every status page I’ve ever seen. It takes them a while to realize there’s really a problem and then to update the page. One day these things will be automated, but until then, I wouldn’t expect more of Cloudflare than any other provider. What’s more concerning to me is that now we’ve had AWS, Azure, and CloudFlare (and CliudFlare twice) go down recently. My gut says: 1. developers an…

Thing is, these things are automated... Internally.

Which makes it feel that much more special when a service provides open access to all of the infrastructure diagnostics, like e.g. https://status.ppy.sh/

Re: Cloudflare outage on December 5, 2025

#342

Post by carl_dr » Fri, Dec 05, 2025, 10:06 PM UTC

XCSme wrote: post #318

al_borland wrote: post #311

Earlier quoted context omitted.

Even if resolution times are equal, there is some comfort in being able to see the problem and make progress on it to feel like you're actively doing something. I work in a large enterprise and we have a team dedicated to managing critical incidents and getting everyone together for a resolution. When a 3rd party vendor is the reason for the outage, those calls are really awkward. It's a bunch of people sitting aroun…

Today a client is having some issue with Zoom because of some artificial rate limits they impose. Their support is not responding, the account can't be used, courses can not be held and there's not much we can do. We already started looking into moving away from Zoom, I suggested self-hosting http://jitsi.org Based on their docs, self-hosting is well supported, and probably a $50-$100 server is more than enough, so a…

Artifical limits because they are on the free plan?

Re: Cloudflare outage on December 5, 2025

#343

Post by autoexec » Fri, Dec 05, 2025, 10:09 PM UTC

evanelias wrote: post #270

autoexec wrote: post #213

Earlier quoted context omitted.

> However, the company is so big that there’s so many different products all shipping at the same time it can be hard to correlate it to your release This kind of thing would be more understandable for a company without hundreds of billions of dollars, and for one that hasn't centralized so much of the internet. If a company has grown too large and complex to be well managed and effective and it's starting to look li…

What "hundreds of billions of dollars"? Cloudflare's annual revenue is around $2 billion, and they are not yet profitable.

That was admittedly hyperbole, but since we're talking about a company with assets and revenue in the billions I'm not sure it matters. The fact remains that a lack of money/resources is not their problem.

Re: Cloudflare outage on December 5, 2025

#344

Post by OneDeuxTriSeiGo » Fri, Dec 05, 2025, 10:11 PM UTC

jokoon wrote: post #339

I still don't understand what is cloudflare's business model, yet they manage to make news. I don't see how their main product is ddos protection, yet cloudflare goes down for some reason. This company makes zero sense to me.

Cloudflare protects against DDOS but also various forms of malicious traffic (bots, low reputation IP users, etc) and often with a DDOS or similar attacks, it's better to have the site go down from time to time than for the attackers to hammer the servers behind cloudflare and waste mass amounts of resources.

i.e. it's the difference between "site goes down for a few hours every few months" and "an attacker slammed your site, and through in on-demand scaling or serverless component cloud fees blew your entire infrastructure budget for the year.

Doubly so when your service is part of a larger platform and attacks on your service risk harming your reputation for the larger platform.

Re: Cloudflare outage on December 5, 2025

#345

Post by j45 » Fri, Dec 05, 2025, 10:15 PM UTC

Curious if there isn't a way to ingest the incoming traffic at scale, but route it to a secondary infrastructure to make sure it's resolving correctly, before pushing it to production?

Re: Cloudflare outage on December 5, 2025

#346

Post by reassess_blind » Fri, Dec 05, 2025, 10:16 PM UTC

hrimfaxi wrote: post #33

Having their changes fully propagate within 1 minute is pretty fantastic.

Why wasn’t the rollback fixed within the second minute after they saw the 500s?

Re: Cloudflare outage on December 5, 2025

#347

Post by XCSme » Fri, Dec 05, 2025, 10:17 PM UTC

carl_dr wrote: post #342

XCSme wrote: post #318

Earlier quoted context omitted.

Today a client is having some issue with Zoom because of some artificial rate limits they impose. Their support is not responding, the account can't be used, courses can not be held and there's not much we can do. We already started looking into moving away from Zoom, I suggested self-hosting http://jitsi.org Based on their docs, self-hosting is well supported, and probably a $50-$100 server is more than enough, so a…

Artifical limits because they are on the free plan?

Artifical limits, because they have 40 paid licenses that they can not use, because of a non-disclosed assignment limit that is NOT mentioned in the pricing page nor in the ToS.

A lot of people are angry about this, and I think it's borderline illegal: https://devforum.zoom.us/t/you-have-exceeded-the-limit-of-li...

You pay for something, and you can't use it.

Re: Cloudflare outage on December 5, 2025

#348

Post by mewpmewp2 » Fri, Dec 05, 2025, 10:18 PM UTC

notepad0x90 wrote: post #264

vlovich123 wrote: post #222

Earlier quoted context omitted.

> Control-plane and data-plane should be separate They are separate. > a react patch shouldn't affect traffic forwarding. If you can’t even bother to read the blog post maybe you shouldn’t be so confident in your own analysis of what should and shouldn’t have happened? This was a configuration change to change the buffered size of a body from 256kb to 1mib. The ability to be so wrong in so few words with such confide…

You really should take some of your pill. > Instead, it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability disclosed this week in React Server Components. > Unfortunately, in our FL1 version of our proxy, under certain circumstances, the second change of turning off our WAF rule testing tool caused an error state that resulted in 500 HT…

To be clear:

1. There is an active vulnerability unrelated to Cloudflare where React/Next.JS can be abused via a malicious payload. The payload could be up to 1MB.

2. Cloudflare had buffer size that wasn't enough to prevent that payload from being sent to the Customer of the Cloudflare.

3. Cloudflare to protect their customers wanted to increase the buffer size to 1MB.

4. Internal Testing Tool wasn't able to handle change to 1MB and started failing.

5. They wanted to stop Internal Testing Tool from failing, but the Internal Testing Tool required disabling a ruleset which an existing system was depending on (due to a long existing bug). This caused the wider incident.

It does seem to be like a mess in the sense that in order to stop internal testing tool from failing they had to endanger things globally in production, yes. It looks like legacy, tech debt mess.

It seems like bad decisions done in the past though.

Re: Cloudflare outage on December 5, 2025

#349

Post by rezonant » Fri, Dec 05, 2025, 10:22 PM UTC

TechniKris wrote: post #341

cnnlives265 wrote: post #315

Earlier quoted context omitted.

This is the case with just about every status page I’ve ever seen. It takes them a while to realize there’s really a problem and then to update the page. One day these things will be automated, but until then, I wouldn’t expect more of Cloudflare than any other provider. What’s more concerning to me is that now we’ve had AWS, Azure, and CloudFlare (and CliudFlare twice) go down recently. My gut says: 1. developers an…

Thing is, these things are automated... Internally. Which makes it feel that much more special when a service provides open access to all of the infrastructure diagnostics, like e.g. https://status.ppy.sh/

Nice! Didn't know you could make a Datadog dashboard public like that!

Re: Cloudflare outage on December 5, 2025

#350

Post by seanparsons » Fri, Dec 05, 2025, 10:27 PM UTC

"This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur." It's starting to sound like a broken record at this point, languages are still seen as equal and as a result, interchangeable.