Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Probably because this case was something more akin to an assert than an error check.
Cloudflare outage on November 18, 2025 post mortem
211–220 of 953 posts
Re: Cloudflare outage on November 18, 2025 post mortem
#212Wow, crazy disproportional drop in the stock price… good buying opportunity for $NET.
Cloudflare is very cheap at these prices.
Re: Cloudflare outage on November 18, 2025 post mortem
#213Earlier quoted context omitted.
How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…
Exception is hidden control flow, where as error values are not. That is the main reason why zig doesn’t have exceptions.
Re: Cloudflare outage on November 18, 2025 post mortem
#214This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…
The real issue is further up the chain where the malformed feature file got created and deployed without better checks.
Re: Cloudflare outage on November 18, 2025 post mortem
#215Re: Cloudflare outage on November 18, 2025 post mortem
#216I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open. Of course, some…
So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?
Re: Cloudflare outage on November 18, 2025 post mortem
#217Earlier quoted context omitted.
How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…
There are many many pages of text discussing this topic, but having programmed in both styles, exceptions make it too easy for programmer to simply ignore them. Errors as values force you to explicitly handle it there, or toss it up the stack. Maybe some other languages have better exception handling but in Python it’s god awful. In big projects you can basically never know when or how something can fail.
With return values, you can trivially ignore an exception.
let _ = fs::remove_file("file_doesn't_exist");
or
value, error = some_function()
// carry on without doing anything with error
In the wild, I've seen far more ignoring return errors, because of the mechanical burden of having type handling at every function call.This is backed by decades of writing libraries. I've tried to implement libraries without exceptions, and was my admittedly cargo-cult preference long ago, but ignoring errors was so prevalent among the users of all the libraries that I now always include a "raise" type boolean that defaults to True for any exception that returns an error value, to force exceptions, and their handling, as default behavior.
> In big projects you can basically never know when or how something can fail.
How is this fundamentally different than return value? Looking at a high level function, you can't know how it will fail, you just know it did fail, from the error being bubbled up through the returns. The only difference is the mechanism for bubbling up the error.
Maybe some water is required for this flame war. ;)
Re: Cloudflare outage on November 18, 2025 post mortem
#218While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language. They are escape hatches. Without those your language would never take off. But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to…
Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do.
If panicking is guaranteed because of some input mistake to the system your failure is in testing.
Re: Cloudflare outage on November 18, 2025 post mortem
#219"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…
I mean, that would require a postmortem from statuspage.io right? Is that a service operated by cloudflare?
Re: Cloudflare outage on November 18, 2025 post mortem
#220This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does.
Yesterday with Cloudflare down half the sites I tried gave me nothing but errors.
The internet is dead.