Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

151–160 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#151

Post by acdha »

olivia-banks wrote: post #99
tptacek wrote: post #69

Earlier quoted context omitted.

What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.

What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than…

The way I’ve seen this on a few older systems was that they always keep the previous configuration around so it can switch back. The logic is something like this:

1. At startup, load the last known good config.

2. When signaled, load the new config.

3. When that passes validation, update the last-known-good pointer to the new version.

That way something like this makes the crash recoverable on the theory that stale config is better than the service staying down. One variant also recorded the last tried config version so it wouldn’t even attempt to parse the latest one until it was changed again.

For Cloudflare, it’d be tempting to have step #3 be after 5 minutes or so to catch stuff which crashes soon but not instantly.

Re: Cloudflare outage on November 18, 2025 post mortem

#152

Post by Dylan16807 »

Rohansi wrote: post #144
SchwKatze wrote: post #111

Earlier quoted context omitted.

Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.

> when you do unwrap() you're saying the Result should NEVER fail Returning a Result by definition means the method can fail.

> Returning a Result by definition means the method can fail.

No more than returning an int by definition means the method can return -2.

Re: Cloudflare outage on November 18, 2025 post mortem

#154

Post by tptacek »

otterley wrote: post #148
tptacek wrote: post #139

Earlier quoted context omitted.

Reframe this problem: instead of bot rules being propagated, it's the enrollment of a new customer or a service at an existing customer --- something that must happen at Cloudflare several times a second. Does it still make sense to you to think about that in terms of "pushing new configuration to prod"?

Those aren't the facts before us. Also, CRUD operations relating to a specific customer or user tend not to cause the sort of widespread incidents we saw today.

They're not, they're a response to your claim that "state" and "configuration" are indifferentiable.

Re: Cloudflare outage on November 18, 2025 post mortem

#155

Post by tptacek »

otterley wrote: post #146
tptacek wrote: post #134

Earlier quoted context omitted.

They made the classic distributed systems mistake and actually did something. Never leap to thing-doing!

If they're going to yeet configs into production, they ought to at least have plenty of mitigation mechanisms, including canary deployments and fault isolation boundaries. This was my primary point at the root of this thread. And I hope fly.io has these mechanisms as well :-)

We've written at long, tedious length about how hard this problem is.

Re: Cloudflare outage on November 18, 2025 post mortem

#156

Post by mercnz »

jijji wrote: post #109

this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about

i think that is data rather than code which is where it falls short, in a way you need stringent code and more safeguarded code; it's like if everyone sends you 64k posts as that's all your proxy layer lets in, someone checked sending 128kb and it gave an error before reaching your app - and then someone sends 128kb and the proxy layer has changed - and your app crashes as it was more than 64kb and your app had an assert against that. to actually track issues with erraneous data that overflows well and stuff isn't so much code test but more like fuzz testing, brute force testing etc. which i think people should do; but that's more like we need strong test networks, and also those test networks may need to be more internet like to reflect real issues too, so the whole testing infrastructure in itself becomes difficult to get right - like they have their own tunneling system etc, they could segregate some of their servers and make a test system with better error diagnosis etc potentially. but to my mind, if they had better error propogation back that really identified what was happening and where then that would be a lot better in general. sure, start doing that on a test network. this is something i've beeen tihnking about in general - i made a simple rpc system for being able to send real time rust tracing logs (it allows to just use the normal tracing framework and use a thin rpc layer) back from multiple end servers but that's mostly for granular debugging. i've never quite understood why systems like systemd-journald aren't more network centric when they're going to be big and complex kitchensink approaches - apparently there's dbus support, but to my mind something inbetween debugging level of code and warning/info. like even if it's doing things like 1/20 of log info it's too much volume if things like large files getting close to limits is increasing etc and we can see this as things run, and can see if it's localised or common etc it'd help have more resilient systems. something may already exist in this line but i didn't come across anything in a reasonably passive way - i mean there's debugging tools like dtrace etc that have been around for ages.

Re: Cloudflare outage on November 18, 2025 post mortem

#157

Post by otterley »

tptacek wrote: post #155
otterley wrote: post #146

Earlier quoted context omitted.

If they're going to yeet configs into production, they ought to at least have plenty of mitigation mechanisms, including canary deployments and fault isolation boundaries. This was my primary point at the root of this thread. And I hope fly.io has these mechanisms as well :-)

We've written at long, tedious length about how hard this problem is.

Have a link?

Re: Cloudflare outage on November 18, 2025 post mortem

#158

Post by tptacek »

otterley wrote: post #157
tptacek wrote: post #155

Earlier quoted context omitted.

We've written at long, tedious length about how hard this problem is.

Have a link?

Most recently, a few weeks ago (but you'll find more just a page or two into the blog):

https://fly.io/blog/corrosion/

Re: Cloudflare outage on November 18, 2025 post mortem

#159

Post by danielheath »

dist1ll wrote: post #149
wrs wrote: post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

> every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this. How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.

I mean... yeah, in general. That's what iterators are for.

Re: Cloudflare outage on November 18, 2025 post mortem

#160

Post by otterley »

tptacek wrote: post #158
otterley wrote: post #157

Earlier quoted context omitted.

Have a link?

Most recently, a few weeks ago (but you'll find more just a page or two into the blog): https://fly.io/blog/corrosion/

It's great that you're working on regionalization. Yes, it is hard, but 100x harder if you don't start with cellular design in mind. And as I said in the root of the thread, this is a sign that CloudFlare needs to invest in it just like you have been.
Post reply on HN