Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

561–570 of 601 posts

Re: Cloudflare outage on December 5, 2025

#561

Post by psychoslave »

hunter2_ wrote: post #554
esrauch wrote: post #549

Earlier quoted context omitted.

Ah ok, it is true that if there's a lot of fungible offerings that worse but uncorrelated uptime can be more robust. I think the question then is how much of the Internet has fungible alternatives such that uncorrelated downtime can meaningfully be less impact. If you have a "to buy" shopping list, the existence of alternative shopping list products doesn't help you, when the one you use is down it's just down, the s…

The restaurant-next-door analogy, representing fungibility, isn't quite right. If BofA is closed and you want to do something in person with them, you can't go to an unrelated bank. If Spotify goes down for an hour, you're not likely to become a YT Music subscriber as a stopgap even though they're somewhat fungible. You'll simply wait, and the question is: can I shuffle my schedule instead of elongating it? A better…

If utility power outage is put on the table, then the analogy is almost everyone solely relying on the same grid, in contrast with being wired to a large set of independent providers or even using their own local solar panel or whatever autonomous energy source.

Re: Cloudflare outage on December 5, 2025

#562

Post by noname120 »

kbolino wrote: post #357
mikkupikku wrote: post #323

Earlier quoted context omitted.

Management doesn't like when things like this are automated. They want to "manage" the outage/production/etc numbers before letting them out.

There's no sweet spot I've found. I don't work for Cloudflare but when I did have a status indicator to maintain, you could never please everyone. Users would complain when our system was up but a dependent system was down, saying that our status indicator was a lie. "Fixing" that by marking our system as down or degraded whenever a dependent system was down led to the status indicator being not green regularly, caus…

> whenever a dependent system was down led to the status indicator being not green regularly, causing us to unfairly develop a reputation as unreliable (most broken dependencies had limited blast radius)

You are responsible of your dependencies, unless they are specific integrations. Either switch to more reliable dependencies or add redundancy so that you can switch between providers when any is down.

Re: Cloudflare outage on December 5, 2025

#563

Post by mikkupikku »

colechristensen wrote: post #361
mikkupikku wrote: post #330

Earlier quoted context omitted.

It tracks with my experience in software quality engineering. Asked to find problems with something already working well in the field. Dutifully find bugs/etc. Get told that it's working though so nobody will change anything. In dysfunctional companies, which is probably most of them, quality engineering exists to cover asses, not to actually guide development.

It is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.

The way it always seemed to go for me, when I was in that role, is the product is already complete, development is done, you're handed all the tests/etc that the disinterested developers care to give you, and you're told to make those tests presentable and robust, and increase test coverage. The process of doing that inevitably uncovers issues, but nobody cares because the thing is already done and working, so what was the point of any of it? The point was just to check off a box. At companies like this, the role is bullshit work.

Re: Cloudflare outage on December 5, 2025

#564

Post by chippiewill »

ifwinterco wrote: post #495
w10-1 wrote: post #166

Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of “execute”. I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation? It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineeri…

It's weird reading these reports because they don't seem to test anything at all (or at least there's very little mention of testing). Canary deployment, testing environments, unit tests, integration tests, anything really? It sounds like they test by merging directly to production but surely they don't

The problem is that Cloudflare do incremental rollouts and loads of testing for _code_. But they don't do the same thing for configuration - they globally push out changes because they want rapid response.

It's still a bit silly though, their claimed reasoning probably doesn't really stack up for most of their config changes - I don't see it to be that likely that a 0.1->1->10->100 rollout over the period of 10 minutes would be a catastrophically bad idea for them for _most_ changes.

And to their credit, it does seem they want to change that.

Re: Cloudflare outage on December 5, 2025

#565

Post by cinericius »

mxpxrocks10 wrote: post #533

First, what Cloudflare does is hard and I want to start with that. That being said, I think it’s worth a discussion. How much of the last 3 outages were because of the JGC (the former CTO) retiring and Dane taking over? Did JGC have a steady hand that’s missing? Or was it just time for outages that would have happened anyway? Dane has maintained a culture of transparency which is fantastic, but did something get inje…

Looking at Dane's career history on LinkedIn, it appears that he has only ever been in product and some variety of manager, and his degree is in 'Engineering Management System'. It's an odd choice given that the previous two CTOs (Lee and John) were extremely technical and how core technology is to Cloudflare.

As with any organisation where the CTO is not technical, there will be someone who the 'CTO' has to ask to understand technical situations. In my opinion, that person being asked is the real CTO, for any given situation.

Re: Cloudflare outage on December 5, 2025

#566

Post by chippiewill »

ryandvm wrote: post #247
lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

Indeed. AWS too. I feel like the cloud hosting companies have lost the plot. "They can provide better uptime than us" is the entire rationale that a lot of small companies have when choosing to run everything in the cloud. If they cost more AND they're less reliable, what exactly is the reason to not self host?

Capex vs Opex and scale-out.

For a start-up it's much easier to just pay the Cloud tax than it is to hire people with the appropriate skill sets to manage hardware or to front the cost.

Larger companies on the other hand? Yeah, I don't see the reason to not self host.

Re: Cloudflare outage on December 5, 2025

#567

Post by kccqzy »

nish__ wrote: post #544
kccqzy wrote: post #207

Earlier quoted context omitted.

Config updates are not so clear cut from code changes. Once I worked with a team in the anti-abuse space where the policy is that code deployments must happen over 5 days and config updates can take a few minutes. Then an engineer on the team argued that deploying new Python code doesn’t count as a code change because the CPython interpreter did not change; it didn’t even restart. And indeed given how dynamic Python…

lol man... If your "config" is Turing-complete, that's a code change. Full stop. Bro's just lazy.

Yeah sure. Now tell the people who are designing config languages to make them both powerful and non-Turing-complete. You see, it is a very hard problem for a powerful language to be non-Turing-complete.

Re: Cloudflare outage on December 5, 2025

#569

Post by kortilla »

locknitpicker wrote: post #514
fallous wrote: post #470

Earlier quoted context omitted.

"My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting. Outsourcing that single point of failure doesn't cure my design of that flaw, especially when that architecture's intended use-case is to provide redundancy and fault-tolerance. The problem with pursuing efficiency as the primary value prop is that you will necessarily end up with a brittle result.

> "My architecture depends upon a single point of failure" is a great way to get laughed out of a design meeting. This is a simplistic opinion. Claiming services like Cloudflare are modeled as single points of failure is like complaining that your use of electricity to power servers is a single point of failure. Cloudflare sells a global network of highly reliable edge servers running services like caching, firewall,…

You do know that data centers use backup generators because electricity is a single point of failure right? They even have multiple power supplies plugged into different circuits.

Re: Cloudflare outage on December 5, 2025

#570

Post by kortilla »

morpheos137 wrote: post #415
odie5533 wrote: post #307

Earlier quoted context omitted.

> It's not like anything behind cloudflare is "mission critical" in the sense that lives are at stake or even a huge amount of money is at stake. Yes, there are lots of mission critical systems that use cloudflare and lives and huge amounts of money are at stake.

Can you provide an example of lives being at stake because of a cloud flare outage?

https://creators.spotify.com/pod/profile/epicompliance/episo...
Post reply on HN