Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

281–290 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#281
post #86

Earlier quoted context omitted.

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic. It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used ca…

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.)

Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is expected, (eof, broken socket, file not found,) you either have to use an unnatural return type or accept the performance penalty of the panic that happens when you "throw."

In Rust, the stack trace happens at panic (unwrap), which is when the error isn't handled. IE, it's not when the file isn't found, it's when the error isn't handled.

Re: Cloudflare outage on November 18, 2025 post mortem

#282

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…

One feature failing like this should probably log the error and fail closed. It shouldn't take down everything else in your big proxy that sits in front of your entire business.

Re: Cloudflare outage on November 18, 2025 post mortem

#283
post #39

Earlier quoted context omitted.

Is it actually consul-template? (I have post-consul-template stress disorder).

I'd love to hear any commentary on Consul if anyone else has it.

I think Consul is great, for what it's worth; we were just abusing it.

https://fly.io/blog/a-foolish-consistency/

https://fly.io/blog/corrosion/

Re: Cloudflare outage on November 18, 2025 post mortem

#284
post #58

Earlier quoted context omitted.

This wild `unwrap()` kinda took me aback as well. Someone really believed in themselves writing this. :)

They only recently rewrote their core in Rust ( https://blog.cloudflare.com/20-percent-internet-upgrade/ ) -- given the newness of the system and things like "Over 100 engineers have worked on FL2, and we have over 130 modules" I won't be surprised for further similar incidents.

The irony of a rust rewrite taking down the internet is not lost on me.

Re: Cloudflare outage on November 18, 2025 post mortem

#285
post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

[flagged]

Why give this sort of content more visibility/reach?

I'm sure that's not your intent, so I hope my comment gives you an opportunity to reflect on the effects of syndicating such stupidity, no matter what platform it comes from.

Re: Cloudflare outage on November 18, 2025 post mortem

#286
post #73

Earlier quoted context omitted.

They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy. I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more…

I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue. Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.

I think the idea is to ship the sqlite database around.

It’s not a terrible idea, in that you can test the exact database engine binary in CI, and it’s (by definition) not a single point of failure.

Re: Cloudflare outage on November 18, 2025 post mortem

#287
post #277

Earlier quoted context omitted.

If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for inter…

> graph-heavy code Could you share some more details, maybe one fully concrete scenario? There are lots of techniques, but there's no one-size-fits-all solution.

Sure, these days I'm mostly working on a few compilers. Let's say I want to make a fixed-size SSA IR. Each instruction has an opcode and two operands (which are essentially pointers to other instructions). The IR is populated in one phase, and then lowered in the next. During lowering I run a few peephole and code motion optimizations on the IR, and then do regalloc + asm codegen. During that pass the IR is mutated and indices are invalidated/updated. The important thing is that this phase is extremely performance-critical.

Re: Cloudflare outage on November 18, 2025 post mortem

#288

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

[flagged]

Re: Cloudflare outage on November 18, 2025 post mortem

#289
kudos to getting this blog post out so fast, it’s well written and is appreciated.

i’m a little confused on how this was initially confused for an attack though?

is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the " error mapping that could have initially pointed you guys towards that over an attack.

also a bit taken aback that .unwrap()’s are ever allowed within such an important context.

would appreciate some insight!

Re: Cloudflare outage on November 18, 2025 post mortem

#290
post #205
post #143

Earlier quoted context omitted.

At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases. If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish ther…

> make them look like a GIANT EYESORE React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED comes to mind. I did have to reach to this before, but it certainly works for keeping this out of example code and other things like reading other implementations without the danger being very apparent. At some point it was renamed to __CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE which is much less fun.

[deleted]
Post reply on HN