Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on December 5, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

331–340 of 601 posts

Re: Cloudflare outage on December 5, 2025

#331

Post by al_borland »

XCSme wrote: post #318
al_borland wrote: post #311

Earlier quoted context omitted.

Even if resolution times are equal, there is some comfort in being able to see the problem and make progress on it to feel like you're actively doing something. I work in a large enterprise and we have a team dedicated to managing critical incidents and getting everyone together for a resolution. When a 3rd party vendor is the reason for the outage, those calls are really awkward. It's a bunch of people sitting aroun…

Today a client is having some issue with Zoom because of some artificial rate limits they impose. Their support is not responding, the account can't be used, courses can not be held and there's not much we can do. We already started looking into moving away from Zoom, I suggested self-hosting http://jitsi.org Based on their docs, self-hosting is well supported, and probably a $50-$100 server is more than enough, so a…

It's interesting to see Comcast is using that. I would have expected them to go with the mainstream vendors.

Re: Cloudflare outage on December 5, 2025

#332

Post by SoftTalker »

markus_zhang wrote: post #177
lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

TBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic. I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy…

Yeah, I'd guess I average a power drop once a month or so at home. Never calculated the nines of uptime average, but it's not that infrequent.

I know when I need to reset the clock on my microwave oven.

Re: Cloudflare outage on December 5, 2025

#334

Post by mewpmewp2 »

jamesog wrote: post #257
crote wrote: post #173

Earlier quoted context omitted.

Is a roll back even possible at Cloudflare's size? With small deployments it usually isn't too difficult to re-deploy a previous commit. But once you get big enough you've got enough developers that half a dozen PRs will have been merged since the start of the incident and now. How viable is it to stop the world, undo everything, and start from scratch any time a deployment causes the tiniest issues? Realistically th…

Disclosure: Former Cloudflare SRE. The short answer is "yes" due to the way the configuration management works. Other infrastructure changes or service upgrades might get undone, but it's possible. Or otherwise revert the commit that introduced the package bump with the new code and force that to rollout everywhere rather than waiting for progressive rollout. There shouldn't be much chance of bringing the system to a…

> service upgrades might get undone, but it's possible.

But who knows what issues might reverting other team's stuff bring?

Re: Cloudflare outage on December 5, 2025

#335

Post by tetha »

uyzstvqs wrote: post #122

What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario. Only after that do you use gradual deployment, with a big red oopsie button which immediately roll…

This is kinda what I'm thinking. We're absolutely not at the scale Cloudflare is at.

But we run software and configuration changes through three tiers - first stage for the dev-team only, second stage with internal customers and other teams depending on it for integration and internal usage -- and finally production. Some teams have also split production into different rings depending on the criticality of the customers and the number of customers.

This has lead to a bunch of discussions early on, because teams with simpler software and very good testing usually push through dev and testing with no or little problem. And that's fine. If you have a track record of good changes, there is little reason to artificially prolong deployment in dev and test just because. If you want to, just go through it in minutes.

But after a few spicy production incidents, even the better and faster teams understood and accepted that once technical velocity exists, actual velocity is a choice, or a throttle if you want an analogy.

If you do good, by all means, promote from test to prod within minutes. If you fuck up production several times in a row and start threatening SLAs, slow down, spend more resources on manual testing and improving automated testing, give changes time to simmer in the internally productive environment, spend more time between promotions from production ring to production ring.

And this is on top of considerations of e.g. change risk. Some frontend-only application can move much faster than the PostgreSQL team, because one rollback is a container restart, and the other could be a multi-hour recovery from backups.

Re: Cloudflare outage on December 5, 2025

#336

Post by parchley »

bostik wrote: post #327
testplzignore wrote: post #179

Earlier quoted context omitted.

> They really need to figure out a way to correlate global configuration changes to the errors they trigger as fast as possible. This is what jumped out at me as the biggest problem. A wild west deployment process is a valid (but questionable) business decision, but if you do that then you need smart people in place to troubleshoot and make quick rollback decisions. Their timeline: > 08:47: Configuration change deplo…

> 2 minutes for their automated alerts to fire is terrible I take exception to that, to be honest. It's not desirable or ideal, but calling it "terrible" is a bit ... well, sorry to use the word ... entitled. For context, I have experience running a betting exchange. A system where it's common for a notable fraction of transactions in a medium-volume event to take place within a window of less than 30 seconds. Vast m…

> At Cloudflare's scale, I would not be surprised if they require three consecutive events to trigger an alert.

Sorry but that’s a method you use if you serve 100 requests per second, not when you are at Cloudflare scale. Cloudflare easily have big enough volume that this problem would trigger an instant change in a monitorable failure rate.

Re: Cloudflare outage on December 5, 2025

#338

Post by lionkor »

SoftTalker wrote: post #332
markus_zhang wrote: post #177

Earlier quoted context omitted.

TBF, it depends on the number of outages locally. In my area it is one outage every thunderstorm/snow storm, so unfortunately the up time of my laptop, even with the help of a large, portable battery charging station (which can charge multiple laptops at the same time), is not optimistic. I sometimes fancy that I could just take cash, go into the wood, build a small solar array, collect & cleanse river water, and buy…

Yeah, I'd guess I average a power drop once a month or so at home. Never calculated the nines of uptime average, but it's not that infrequent. I know when I need to reset the clock on my microwave oven.

99.9 is like 9 hours of downtime a year.

Re: Cloudflare outage on December 5, 2025

#339

Post by jokoon »

I still don't understand what is cloudflare's business model, yet they manage to make news.

I don't see how their main product is ddos protection, yet cloudflare goes down for some reason.

This company makes zero sense to me.

Re: Cloudflare outage on December 5, 2025

#340

Post by toomuchtodo »

ryandvm wrote: post #247
lionkor wrote: post #101

Cloudflare is now below 99.9% uptime, for anyone keeping track. I reckon my home PC is at least 99.9%.

Indeed. AWS too. I feel like the cloud hosting companies have lost the plot. "They can provide better uptime than us" is the entire rationale that a lot of small companies have when choosing to run everything in the cloud. If they cost more AND they're less reliable, what exactly is the reason to not self host?

> If they cost more AND they're less reliable, what exactly is the reason to not self host?

Shifting liability. You're paying someone else for it to be their problem, and if everyone does it, no one will take flak for continuing to do so. What is the average tenure of a CIO or decision maker electing to move to or remain at a cloud provider? This is why you get picked to talk on stage at cloud provider conferences.

(have been in the meetings where these decisions are made)

Post reply on HN