Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

381–390 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#381

Post by perching_aix »

sanjitb wrote: post #372

cloudflare: > Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. also cloudflare: > The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part…

I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing.

Cloudflare's status page: https://www.cloudflarestatus.com/

Cloudflare Dashboard: https://dash.cloudflare.com/

Re: Cloudflare outage on November 18, 2025 post mortem

#382

Post by samdoesnothing »

yoyohello13 wrote: post #116

People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue. I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.

The reason why people are criticizing is because Rust evangelicals say stuff like "if it compiles it works" or talk about how Rust's type system is so much better than other languages that it catches logic errors like this. You won't see Go or Java developers making such strong claims about their preferred languages.

Re: Cloudflare outage on November 18, 2025 post mortem

#383

Post by vlovich123 »

Feathercrown wrote: post #344
vlovich123 wrote: post #247

Earlier quoted context omitted.

Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constit…

Incremental config changes sounds like it could lead to a LOT of bugs

Incremental in terms of 1% of the fleet using it, then 5% etc. this is standard course.

Another option is to make sure that config changes that fail to parse continue using the old config instead of resulting in an unusable service.

Re: Cloudflare outage on November 18, 2025 post mortem

#384

Post by samdoesnothing »

ulfw wrote: post #220

The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al. This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Az…

It's not that deep, if AWS or Cloudflare suddenly disappeared sites would move to different hosts, it wouldn't mean the internet would die.

Re: Cloudflare outage on November 18, 2025 post mortem

#385

Post by aw1621107 »

tonyhart7 wrote: post #365
aw1621107 wrote: post #337

Earlier quoted context omitted.

> same like chess, engine is better than human grandmaster because its solvable math field Might be worth noting that your description of chess is slightly incorrect. Chess technically isn't solved in the sense that the optimal move is known for any arbitrary position is known; it's just that chess engines are using what amounts to a fancy brute force for most of the game and the combination of hardware and search al…

No ?????? because these thing called BEST MOVE and BAD MOVE there in chess "chess engines are still capable of making mistakes", I'm sorry no inaccurate yes but not mistake

> because these thing called BEST MOVE and BAD MOVE there in chess

The thing is that there is no known general objective criteria for "best" and "bad" moves. The best we have so far is based on engine evaluations, but as I said before that is because chess engines are better at searching the board's state space than humans, not because chess engines have solved chess in the mathematical sense. Engines are quite capable of misevaluating positions, as demonstrated quite well by the Top Chess Engine Championship [0] where one engine thinks it made a good move while the other thinks that move is bad, and this is especially the case when resources are limited.

The closest we are to solving chess are via tablebases, which are far from covering the entire state space and are basically as much of an exemplar of pure brute force as you can get.

> "chess engines are still capable of making mistakes", I'm sorry no

If you think chess engines are infalliable, then why does the Top Chess Engine Championship exist? Surely if chess engines could not make mistakes they would always agree on a position's evaluation and what move should be made, and therefore such an exercise would be pointless?

> inaccurate yes but not mistake

From the perspective to attaining perfect play an inaccuracy is a mistake.

[0]: https://en.wikipedia.org/wiki/Top_Chess_Engine_Championship

Re: Cloudflare outage on November 18, 2025 post mortem

#386

Post by vlovich123 »

kibwen wrote: post #265
hedora wrote: post #186

Earlier quoted context omitted.

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

I think the parent is implying that the panic should be "caught" via a supervisor process, Erlang-style, rather than implying the literal use of `catch_unwind` to resume within the same process.

Supervisor is the brutalist way. But catch_unwind may be needed for perf and other reasons.

But ultimately it’s not the panic that’s the problem but a failure to specify how panics within FL2 layers should be handled; each layer is at least one team and FL2’s job is providing a safe playground for everyone to safely coexist regardless of the misbehavior of any single component

But as always such failures are emblematic of multiple things going wrong at once. You probably want to end up using both catch_unwind for the typical case and the supervisor for the case where there’s a segfault in some unsafe code you call or native library you invoke.

I also mention the fundamental tension of do you want to fail open or closed. Most layers should probably fail open. Some layers (eg auth) it’s safer to fail closed.

Re: Cloudflare outage on November 18, 2025 post mortem

#388

Post by kccqzy »

antonvs wrote: post #302
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

> This is textbook "parse, don't validate" anti-pattern. How so? “Parse, don’t validate” implies converting input into typed values that prevent representation of invalid state. But the parsing still needs to be done correctly. An unchecked unwrap really has nothing to do with this.

GP completely misunderstands “parse, don’t validate” and also calls it an anti-pattern. GP clearly has no idea what this is.

Re: Cloudflare outage on November 18, 2025 post mortem

#389

Post by aw1621107 »

phire wrote: post #374
echelon wrote: post #346

Earlier quoted context omitted.

Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.) I want to ban crates that panic from my dependency chain. The language could really use an extra set of static guarantees around this. I would opt in.

I think I'd prefer a compile-time guarantee. Something that allows me to tag annotate a function (or my whole crate) as "no panic", and get a compile error if the function or anything it calls has a reachable panic. This will allow it to work with many unmodified crates, as long as constant propagation can prove that any panics are unreachable. This approach will also allow crates to provide panicking and non panicki…

I think the most common solution at the moment is dtolnay's no_panic [0]. That has a bunch of caveats, though, and the ergonomics leave something to be desired, so a first-party solution would probably be preferable.

[0]: https://github.com/dtolnay/no-panic

Re: Cloudflare outage on November 18, 2025 post mortem

#390

Post by echelon »

phire wrote: post #374
echelon wrote: post #346

Earlier quoted context omitted.

Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.) I want to ban crates that panic from my dependency chain. The language could really use an extra set of static guarantees around this. I would opt in.

I think I'd prefer a compile-time guarantee. Something that allows me to tag annotate a function (or my whole crate) as "no panic", and get a compile error if the function or anything it calls has a reachable panic. This will allow it to work with many unmodified crates, as long as constant propagation can prove that any panics are unreachable. This approach will also allow crates to provide panicking and non panicki…

Yes, I want that. I also want to be able to (1) statically apply a badge on every crate that makes and meets these guarantees (including transitively with that crate's own dependencies) so I can search crates.io for stronger guarantees and (2) annotate my Cargo.toml to not import crates that violate this, so time isn't wasted compiling - we know it'll fail in advance.

On the subject of this, I want more ability to filter out crates in our Cargo.toml. Such as a max dependency depth. Or a frozen set of dependencies that is guaranteed not to change so audits are easier. (Obviously we could vendor the code in and be in charge of our own destiny, but this feels like something we can let crate authors police.)

Post reply on HN