Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

541–550 of 601 posts

Re: Cloudflare outage on December 5, 2025

#541

Post by rolisz »

philipallstar wrote: post #501
Lamprey wrote: post #432

Earlier quoted context omitted.

It's not "1,000 wasps or 1 dog", it's "1,000 dogs at once, or "1 dog at once, 1,000 different times". Rare but huge and coordinated siege, or a steady and predictable background radiation of small issues. The latter is easier to handle, easier to fix, and much more suvivable if you do fuck it up a bit. It gives you some leeway to learn from mistakes. If you make a mistake during the 1000 dog siege, or if you don't ha…

I don't quite see how this maps onto the situation. The "1000 dog seige" also was resolved very quickly and transparently, so I would say it's actually better than even one of the "1 dog at once"s.

Last week's cloudflare outage was not resolved as quickly...

Re: Cloudflare outage on December 5, 2025

#542

Post by matteocontrini »

jacobgkau wrote: post #288

I noticed this outage last night (Cloudflare 500s on a few unrelated websites). As usual, when I went to Cloudflare's status page, nothing about the outage was present; the only thing there was a notice about the pre-planned maintenance work they were doing for the security issue, reporting that everything was being routed around it successfully.

The status page was updated 6 minutes after the first internal alert was triggered (8:50 -> 8:56:26 UTC), I wouldn't say this is too long.

Re: Cloudflare outage on December 5, 2025

#543

Post by p-e-w »

shermantanktop wrote: post #474
ccakes wrote: post #450

Earlier quoted context omitted.

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…

Maybe worlds can just live without the internet for a few hours. There are likely emergency services dependent on Cloudflare at this point, so I’m only semi serious.

The world dismantled landlines, phone booths, mail order catalogues, fax machines, tens of millions of storefronts, government offices, and entire industries in favor of the Internet.

So at this point no, the world can most definitely not “just live without the Internet”. And emergency services aren’t the only important thing that exists to the extent that anything else can just be handwaved away.

Re: Cloudflare outage on December 5, 2025

#544

Post by nish__ »

kccqzy wrote: post #207
nish__ wrote: post #119

Is it crazy to anyone else that they deploy every 5 minutes? And that it's not just config updates, but actual code changes with this "execute" action.

Config updates are not so clear cut from code changes. Once I worked with a team in the anti-abuse space where the policy is that code deployments must happen over 5 days and config updates can take a few minutes. Then an engineer on the team argued that deploying new Python code doesn’t count as a code change because the CPython interpreter did not change; it didn’t even restart. And indeed given how dynamic Python…

lol man... If your "config" is Turing-complete, that's a code change. Full stop. Bro's just lazy.

Re: Cloudflare outage on December 5, 2025

#545

Post by psychoslave »

mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

That's a reflect of social organisation. Pushing for hierarchical organisation with a few key centralising nodes will also impact business and technological decisions.

See also https://en.wikipedia.org/wiki/Conway%27s_law

Re: Cloudflare outage on December 5, 2025

#546

Post by psychoslave »

esrauch wrote: post #538
ccakes wrote: post #450

Earlier quoted context omitted.

> If an individual site took on the infra challenges themselves, would they achieve better? I don’t think so. The point is that it doesn’t matter. A single site going down has a very small chance of impacting a large number of users. Cloudflare going down breaks an appreciable portion of the internet. If Jim’s Big Blog only maintains 95% uptime, most people won’t care. If BofA were at 95%.. actually same. Most of the…

I'm not sure I follow the argument. If literally every individual site had an uncorrelated 99% uptime, that's still less available than a centralized 99.9% uptime. The "entire Internet" is much less available in the former setup. It's like saying that Chipotle having X% chance of tainted food is worse than local burrito places having 2*X% chance of tainted food. It's true in the lens that each individual event affect…

No it's like saying if one single point of failure in a global food supply chain fails, nobody's going to eat today. And which is in contrast to if some supplier fails to provide a local food truck today their customers will have to go to the restaurant next door.

Re: Cloudflare outage on December 5, 2025

#547

Post by coderjames »

chickensong wrote: post #385
mixedbit wrote: post #366

This is architectural problem, the LUA bug, the longer global outage last week, a long list of earlier such outages only uncover the problem with architecture underneath. The original, distributed, decentralized web architecture with heterogeneous endpoints managed by myriad of organisations is much more resistant to this kind of global outages. Homogeneous systems like Cloudflare will continue to cause global outage…

You're not wrong, but where's the robust architecture you're referring to? The reality of providing reliable services on the internet is far beyond the capabilities of most organizations.

I think it might be a organizational architecture that needs to change.

> However, we have never before applied a killswitch to a rule with an action of “execute”.

> This is a straightforward error in the code, which had existed undetected for many years

So they shipped an untested configuration change that triggered untested code straight to production. This is "tell me you have no tests without telling me you have no tests" level of facepalm. I work on safety-critical software where if we had this type of quality escape both internal auditors and external regulators would be breathing down our necks wondering how our engineering process failed and let this through. They need to rearchitect their org to put greater emphasis on verification and software quality assurance.

Re: Cloudflare outage on December 5, 2025

#548

Post by z00z1 »

winddude wrote: post #438
andrewf wrote: post #381

Earlier quoted context omitted.

Quickly Googling about, a commonly repeated figure is that Akamai served 15% - 30% of Internet traffic in the late 2010's. They probably have less of the market today due to others growing, but they're not a minnow. 2024 revenue figures were $1.669 billion for Cloudflare, and $3.99 billion for Akamai, per Wikipedia.

https://w3techs.com/technologies/overview/proxy , they are tiny compared to CF, their revenue is high because they focus on large enterprise clients.

> 20% of internet traffic passes through CF networks

That does not sound right to me. “20 percent of websites” does not mean “20 percent of traffic.”.

There is no public write-up from Cloudflare that proves “we handle 20% of all Internet traffic.” Cloudflare reports around 295,000 paying customers and more than 30 million Internet properties (20% of the web). So most of their users are on the free plan.

Re: Cloudflare outage on December 5, 2025

#549

Post by esrauch »

psychoslave wrote: post #546
esrauch wrote: post #538

Earlier quoted context omitted.

I'm not sure I follow the argument. If literally every individual site had an uncorrelated 99% uptime, that's still less available than a centralized 99.9% uptime. The "entire Internet" is much less available in the former setup. It's like saying that Chipotle having X% chance of tainted food is worse than local burrito places having 2*X% chance of tainted food. It's true in the lens that each individual event affect…

No it's like saying if one single point of failure in a global food supply chain fails, nobody's going to eat today. And which is in contrast to if some supplier fails to provide a local food truck today their customers will have to go to the restaurant next door.

Ah ok, it is true that if there's a lot of fungible offerings that worse but uncorrelated uptime can be more robust.

I think the question then is how much of the Internet has fungible alternatives such that uncorrelated downtime can meaningfully be less impact. If you have a "to buy" shopping list, the existence of alternative shopping list products doesn't help you, when the one you use is down it's just down, the substitutes cannot substitute on short notice. Obviously for some things there's clear substitutes though, but actually I think "has fungible alternatives" is mostly correlated with "being down for 30 minutes doesn't matter", it seems that the things where you want the one specific site are the ones where availability matters more.

Re: Cloudflare outage on December 5, 2025

#550

Post by dzonga »

before Cloudflare suffered an outage due to React's useEffect, now again trying to mitigate security issues around React Server Pages.

at one point in time - they've to admit this react thing ain't working. & just use classic server rendered pages, since their dashboards are simple toggle controls

Post reply on HN