Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

461–470 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#461

Post by throwaway2037 »

echelon wrote: post #425
lenkite wrote: post #410

Earlier quoted context omitted.

> I want to ban crates that panic from my dependency chain. Which means banning anything that allocates memory and thousands of stdlib functions/methods.

See the immediately preceding sentence. I'm fine with allocation failures. I don't want stupid unwrap()s, improper slice access, or other stupid and totally preventable behavior. There are things inside the engineer's control. I want that to not panic.

Your pair of posts is very interesting to me. Can you share with me: What is your programming environment such that you are "fine with allocation failures"? I'm not doubting you, but for me, if I am doing systems programming with C or C++, my program is doomed if a malloc fails! When I saw your post, I immediately thought: Am I doing it wrong? If I get a NULL back from malloc(), I just terminate with an error message.

Re: Cloudflare outage on November 18, 2025 post mortem

#462

Post by makach »

Excellent write up. Cybersecurity professionals read the story and learn. It’s textbook lesson in post-mortem incident analysis - a mvp for what is expected from us all in a similar situation.

Reputationally this is extremely embarrassing for Cloudflare, but imo they seem to get their feet back on the ground. I was surprised to see not just one, but two apologies to the internet. This just cements how professional and dedicated the Cloudflare team is to ensure stable resilient internet and how embarrassed they must have been.

A reputational hit for sure, but outcome is lessons learned and hopefully stronger resilience.

Re: Cloudflare outage on November 18, 2025 post mortem

#463

Post by rixed »

RagingCactus wrote: post #102

Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate. Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as…

Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire.

The issue here is about the system as a whole not any line of code.

Re: Cloudflare outage on November 18, 2025 post mortem

#464

Post by ChrisMarshallNY »

Pulcinella wrote: post #366
monocularvision wrote: post #358

Earlier quoted context omitted.

So what happens if it ends up being nil? How does your app react? In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace. Silent failure is ultimately terrible for users. Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rath…

Yeah @IBOutlets are generally the one thing that are allowed to be implicitly-unwrapped optionals. They go along with using storyboards & xibs files with Interface Builder. I agree that you really should just crash if you are attempting to access one and it is nil. Either you have done something completely incorrect with regards to initializing and accessing parts of your UI and want to catch that in development, or…

> … you really should just crash if …

See my above/below comment.

A good tool for catching stuff during development, is the humble assert()[0]. We can use precondition()[1], to do the same thing, in ship code.

The main thing is, is to remain in control, as much as possible. Rather than let the PC leave the stack frame, throw the error immediately when it happens.

[0] https://docs.swift.org/swift-book/documentation/the-swift-pr...

[1] https://docs.swift.org/swift-book/documentation/the-swift-pr...

Re: Cloudflare outage on November 18, 2025 post mortem

#467

Post by fredoralive »

jcalvinowens wrote: post #317
AdieuToLogic wrote: post #309

Earlier quoted context omitted.

>> This is the multi-million dollar .unwrap() story. > That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assu…

> Instead, the routine likely should have worked within `Result`. But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same. Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that. But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invar…

A failed config load probably shouldn't be a fatal error if a valid config is already loaded?

Re: Cloudflare outage on November 18, 2025 post mortem

#468

Post by karel-3d »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

As a gopher I never understand why is there so many unwraps in an average rust code.

Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

Re: Cloudflare outage on November 18, 2025 post mortem

#469

Post by branko_d »

gwbas1c wrote: post #281
dehrmann wrote: post #231

Earlier quoted context omitted.

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.) Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is e…

> Exceptions force a panic on all errors

What do you mean?

Exceptions do not force panic at all. In most practical situations, an exception unhandled close to where it was thrown will eventually get logged. It's kind of a "local" panic, if you will, that will terminate the specific function, but the rest of the program will remain unaffected. For example, a web server might throw an exception while processing a specific HTTP request, but other HTTP requests are unaffected.

Throwing an exception does not necessarily mean that your program is suddenly in an unsupported state, and therefore does not require terminating the entire program.

Re: Cloudflare outage on November 18, 2025 post mortem

#470

Post by richardwhiuk »

karel-3d wrote: post #468
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

As a gopher I never understand why is there so many unwraps in an average rust code. Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

Because Go silently gives you zero/null instead
Post reply on HN