Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

711–720 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#711

Post by quotemstr »

frumplestlatz wrote: post #549
brabel wrote: post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality. Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled. If you need the value, you need to handle the case that it’s not available explicitly. Y…

> If you have to use unwrap, your types do not match your problem

The problem starts with Rust stdlib. It panics on allocation failure. You expect Rust programmers to look at stdlib and not imitate it?

Sure, you can try to taboo unwrap(), but 1) it won't work, and 2) it'll contort program design in places where failure really is a logic bug, not a runtime failure, and for which unwrap() is actually appropriate.

The real solution is to go back in time, bonk the Rust designers over the head with a cluebat, and have them ship a language that makes error propagation the default and syntactically marks infallible cleanup paths --- like C++ with noexcept.

Re: Cloudflare outage on November 18, 2025 post mortem

#712

Post by petcat »

echelon wrote: post #636
wongarsu wrote: post #552

Earlier quoted context omitted.

> at least do `expect("damn it, how did this happen")` That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust

We shouldn't be using unwrap() or expect() at all. This is Rust's Null Pointer Exception. unwrap(), expect(), bad math, etc. - this is all caused by lazy Rust developers or Rust developers not utilizing the language's design features. The language should grow the ability to mark this code as dangerous, and we should have static tools to exclude this code from our dependency tree. I don't want some library I use to `u…

How would they plug it? Just deprecate .unwrap and .expect and then remove them from the API completely?

Re: Cloudflare outage on November 18, 2025 post mortem

#713

Post by hu3 »

Klonoar wrote: post #511
uecker wrote: post #376

Earlier quoted context omitted.

While I agree that Rust got it right by being more explicit, a lot of bugs in C/C++ can also easily avoided with good engineering practices. The Rust argument that it is mainly the fault of the programming language with C/C++ was always a huge and unfair exaggeration. Now with this entirely predictable ".unwrap" desaster (in general, not necessarily this exact scenarious), the "no true Rustacean would have put unwrap…

> the "no true Rustacean would have put unwrap in production" The "no unwrap" rule is common in most production codebases. Chill.

Could you point one that is Open source?

Re: Cloudflare outage on November 18, 2025 post mortem

#714

Post by zero_shift »

gwd wrote: post #633
zero_shift wrote: post #610

Earlier quoted context omitted.

But I could screw it up in Go, if I made the same assumptions fvs, err := features.AppendWithNames(..) if err != nil { // this will NEVER break panic(err) } Ultimately I don't think language design can be the sole line of defence against system failures; it can only guide developers to think about error cases

Right, but the point isn't to make errors impossible; the point is to have them be 1) less likely to write, and 2) easier to spot on review. People's biggest complaints about golang's errors: 1. You have to _TYPE_OUT_ what to do on EVERY.SINGLE.ERROR. SOO BOORING! 2. They clutter up the code and make it look ugly. Rust is so much cleaner and more convenient (they say)! Just add ?, or .unwrap()! Well, with ".unwrap()"…

Eh, I'm not convinced.

1. Culturally, using `unwrap` is an omerta to Rust developers in the same way `panic` is an omerta to Go devs;

2. In the Rust projects I've seen there is usually a linter rule forbidding `unwrap` so you can't use it in production

Re: Cloudflare outage on November 18, 2025 post mortem

#716

Post by pornel »

vsgherzi wrote: post #22

Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type. At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect"). The whole point of errors as values is preventing this kind of thing.... It w…

unwrap() is only the most superficial part of the problem. Merely replacing `unwrap()` with `return Err(code)` wouldn't have changed the behavior. Instead of "error 500 due to panic" the proxy would fail with "error 500 due to $code".

Unwrap gives you a stack trace, while retuned Err doesn't, so simply using a Result for that line of code could have been even harder to diagnose.

`unwrap_or_default()` or other ways of silently eating the error would be less catastrophic immediately, but could still end up breaking the system down the line, and likely make it harder to trace the problem to the root cause.

The problem is deeper than an unwrap(), related to handling rollouts of invalid configurations, but that's not a 1-line change.

Re: Cloudflare outage on November 18, 2025 post mortem

#718

Post by kqr »

janpio wrote: post #663
kqr wrote: post #643

One of the remediations listed is > Eliminating the ability for core dumps or other error reports to overwhelm system resources but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.

I understood this to be related to this section: > As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information. (Just above https://blog.cloudflare.com/18-november-202…

Ah, yes, of course. I must have missed that. Thanks!

Re: Cloudflare outage on November 18, 2025 post mortem

#719

Post by echelon »

an_ko wrote: post #698
echelon wrote: post #655

Earlier quoted context omitted.

https://github.com/search?q=unwrap%28%29+language%3ARust&typ... This is sobering. My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness. Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.

I use unwrap a lot, and my most frequent target is unwrapping the result of Mutex::lock. Most applications have no reasonable way to recover from lock poisoning, so if I were forced to write a match for each such use site to handle the error case, the handler would have no choice but to just call panic anyway. Which is equivalent to unwrap, but much more verbose. Perhaps it needs a scarier name, like "assume_ok".

I use locks a lot too, and I always return a Result from lock access. Sometimes an anyhow::Result, but still something to pass up to the caller.

This lets me do logging at minimum. Sometimes I can gracefully degrade. I try to be elegant in failure as possible, but not to the point where I wouldn't be able to detect errors or would enter a bad state.

That said, I am totally fine with your use case in your application. You're probably making sane choices for your problem. It should be on each organization to decide what the appropriate level of granularity is for each solution.

My worry is that this runtime panic behavior has unwittingly seeped into library code that is beyond our ability and scope to observe. Or that an organization sets a policy, but that the tools don't allow for rigid enforcement.

Re: Cloudflare outage on November 18, 2025 post mortem

#720

Post by twoodfin »

thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

The query is surely faulty: Even if this wasn’t a huge distributed database with who-knows-what schemas and use cases, looking up a specific table by its unqualified name is sloppy.

But the architectural assumption that the bot file build logic can safely obtain this operationally critical list of features from derivative database metadata vs. a SSOT seems like a bigger problem to me.

Post reply on HN