ClickHouse db was mentioned. Does this incident raise any critiques about it?
Cloudflare outage on November 18, 2025 post mortem
421–430 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#422Earlier quoted context omitted.
I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing. Cloudflare's status page: https://www.cloudflarestatus.com/ Cloudflare Dashboard: https://dash.cloudflare.com/
thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.
Yes, that was missing (along with the London WARP thing). Other comments mentioned that their status page is an Atlassian Statuspage solution, hosted on AWS CloudFront.
Unclear to me if it's an Atlassian-managed deployment they have, or if it's self-managed, I'm not familiar with Statuspage and their website isn't helping. Though if it's managed, I'm not sure how they can know for sure there's no interdependence. (Though I guess we could technically keep that rabbit hole going indefinitely.)
Re: Cloudflare outage on November 18, 2025 post mortem
#423Earlier quoted context omitted.
What about memory allocation - how will you stop that from panicking ? `Vec::resize` will always panic in Rust. And this is just one example out of thousands in the Rust stdlib. Unless the language addresses no-panic in its governing design or allows try-catch, not sure how you go about this.
That is slowly being addressed, but meanwhile it’s likely you have a reliable upper bound on how much heap your service needs, so it’s a much smaller worry. There are also techniques like up-front or static allocation if you want to make more certain.
Yep and this postmortem details how their proxy modules use static allocation.
Re: Cloudflare outage on November 18, 2025 post mortem
#424Earlier quoted context omitted.
There are even lints for this but people get impatient and just override them or fight for them to no longer be the default. As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.
and people make mistake at some point machine would be better in coding because well machine code is machine instruction task same like chess, engine is better than human grandmaster because its solvable math field coding is no different
[deleted]
Re: Cloudflare outage on November 18, 2025 post mortem
#425Earlier quoted context omitted.
Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.) I want to ban crates that panic from my dependency chain. The language could really use an extra set of static guarantees around this. I would opt in.
> I want to ban crates that panic from my dependency chain. Which means banning anything that allocates memory and thousands of stdlib functions/methods.
See the immediately preceding sentence.
I'm fine with allocation failures. I don't want stupid unwrap()s, improper slice access, or other stupid and totally preventable behavior.
There are things inside the engineer's control. I want that to not panic.
Re: Cloudflare outage on November 18, 2025 post mortem
#426Earlier quoted context omitted.
You misunderstand what Rust’s guarantees are. Rust has never promised to solve or protect programmers from logical or poor programming. In fact, no such language can do that, not even Haskell. Unwrapping is a very powerful and important assertion to make in Rust whereby the programmer explicitly states that the value within will not be an error, otherwise panic. This is a contract between the author and the runtime.…
> Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought I mean thats an unfalsifiable statement, not really fair. C is used to successfully launch spaceships. Whereas we have a real Rust bug that crashed a good portion of the internet for a significant amount of…
Only formal proof languages are immune to such properties. Therefore all languages are poorly designed by your metric.
Consider that the set of possible failures enabled by language design should be as small as possible.
Rust's set is small enough while also being productive. Until another breakthrough in language design as impactful as the borrow checker is invented, I don't imagine more programmers will be able to write such a large amount of safe code.
Re: Cloudflare outage on November 18, 2025 post mortem
#427Did some $300k chief of IT blame it all on some overworked secretary clicking a link in an email they should have run through a filter? Because that’s the MO.
Hopefully that's the case.
Re: Cloudflare outage on November 18, 2025 post mortem
#428This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a n…
Falling back to a generic base configuration in the presence of an incoming invalid config file would probably be a sensible thing to do.
Re: Cloudflare outage on November 18, 2025 post mortem
#429Earlier quoted context omitted.
I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency. Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…
that's very cool, thanks
Re: Cloudflare outage on November 18, 2025 post mortem
#430Earlier quoted context omitted.
No ?????? because these thing called BEST MOVE and BAD MOVE there in chess "chess engines are still capable of making mistakes", I'm sorry no inaccurate yes but not mistake
> because these thing called BEST MOVE and BAD MOVE there in chess The thing is that there is no known general objective criteria for "best" and "bad" moves. The best we have so far is based on engine evaluations, but as I said before that is because chess engines are better at searching the board's state space than humans, not because chess engines have solved chess in the mathematical sense. Engines are quite capab…
"The thing is that there is no known general objective criteria for "best" and "bad" moves."
are you playing chess or not?????? if you playing chess then its oblivious how to differentiate bad move and best move
Yes it is objective, these thing called best move not without reason
"If you think chess engines are infalliable, then why does the Top Chess Engine Championship exist?"
to create better chess engine like what do even talking about here????, are you saying just because there are older bad engine that mean this thing is pointless ????
if you playing chess up to a decent level 1700+ (like me), you know that these argument its wrong and I assure you to learn chess to a decent level
up until that point that you know high level chess is brute force games and therefore solvable math