Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

211–220 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#211
post #21

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Probably because this case was something more akin to an assert than an error check.

In that case, it should probably be expect rather than unwrap, to document why the assertion should never fail.

Re: Cloudflare outage on November 18, 2025 post mortem

#213
post #178

Earlier quoted context omitted.

How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…

Exception is hidden control flow, where as error values are not. That is the main reason why zig doesn’t have exceptions.

I'd categorize them more as "event handlers" than "hidden". You can't know where the execution will go at a lower level, but that's the entire point: you don't care. You put the handlers at the points where you care.

Re: Cloudflare outage on November 18, 2025 post mortem

#214

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a normal thing to do. There's no graceful handling (beyond a nice error message) a program like Nginx could do on a syntax error in its config.

The real issue is further up the chain where the malformed feature file got created and deployed without better checks.

Re: Cloudflare outage on November 18, 2025 post mortem

#216
post #209

I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open. Of course, some…

So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?

Only if they are able to block the siteverify check performed by our backend server. That's not the kind of attack we are trying to mitigate with Turnstile.

Re: Cloudflare outage on November 18, 2025 post mortem

#217
post #178

Earlier quoted context omitted.

How so? An exception is a value that's given the closest, conceptually appropriate, point that was decided to handle the value, allowing you to keep your "happy path" as clean code, and your "exceptional circumstances" path at the level of abstraction that makes sense. It's way less book-keeping with exceptions, since you, intentionally, don't have to write code for that exceptional behavior, except where it makes se…

There are many many pages of text discussing this topic, but having programmed in both styles, exceptions make it too easy for programmer to simply ignore them. Errors as values force you to explicitly handle it there, or toss it up the stack. Maybe some other languages have better exception handling but in Python it’s god awful. In big projects you can basically never know when or how something can fail.

I would claim the opposite. If you don't catch an exception, you'll get a halt.

With return values, you can trivially ignore an exception.

    let _ = fs::remove_file("file_doesn't_exist");

    or

    value, error = some_function()
    // carry on without doing anything with error
In the wild, I've seen far more ignoring return errors, because of the mechanical burden of having type handling at every function call.

This is backed by decades of writing libraries. I've tried to implement libraries without exceptions, and was my admittedly cargo-cult preference long ago, but ignoring errors was so prevalent among the users of all the libraries that I now always include a "raise" type boolean that defaults to True for any exception that returns an error value, to force exceptions, and their handling, as default behavior.

> In big projects you can basically never know when or how something can fail.

How is this fundamentally different than return value? Looking at a high level function, you can't know how it will fail, you just know it did fail, from the error being bubbled up through the returns. The only difference is the mechanism for bubbling up the error.

Maybe some water is required for this flame war. ;)

Re: Cloudflare outage on November 18, 2025 post mortem

#218

While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language. They are escape hatches. Without those your language would never take off. But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to…

This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable.

Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do.

If panicking is guaranteed because of some input mistake to the system your failure is in testing.

Re: Cloudflare outage on November 18, 2025 post mortem

#219
post #29
post #20

"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…

I mean, that would require a postmortem from statuspage.io right? Is that a service operated by cloudflare?

Atlasaian

Re: Cloudflare outage on November 18, 2025 post mortem

#220
The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al.

This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does.

Yesterday with Cloudflare down half the sites I tried gave me nothing but errors.

The internet is dead.

Post reply on HN