Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

511–520 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#511

Post by Klonoar »

uecker wrote: post #376
Klonoar wrote: post #132

Earlier quoted context omitted.

> I don't use Rust, but a lot of Rust people say if it compiles it runs. Do you grok what the issue was with the unwrap, though...? Idiomatic Rust code does not use that. The fact that it's allowed in a codebase says more about the engineering practices of that particular project/module/whatever. Whoever put the `unwrap` call there had to contend with the notion that it could panic and they still chose to do it. It's…

While I agree that Rust got it right by being more explicit, a lot of bugs in C/C++ can also easily avoided with good engineering practices. The Rust argument that it is mainly the fault of the programming language with C/C++ was always a huge and unfair exaggeration. Now with this entirely predictable ".unwrap" desaster (in general, not necessarily this exact scenarious), the "no true Rustacean would have put unwrap…

> the "no true Rustacean would have put unwrap in production"

The "no unwrap" rule is common in most production codebases. Chill.

Re: Cloudflare outage on November 18, 2025 post mortem

#512

Post by cube00 »

ozgrakkurt wrote: post #442
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

Not panicking code is tedious to write. It is not realistic to expect everything to be non panic. There is a reason that panicking exists in the first place. Them calling unwrap on a limit check is the real issue imo. Everything that takes in external input should assume it is bad input and should be fuzz tested imo. In the end, what is the point of having a limit check if you are just unwrapping on it

> Not panicking code is tedious to write.

Using the question mark operator [1] and even adding in some anyhow::context goes a long way to being able to fail fast and return an Err rather then panicking.

Sure you need to handle Results all the way up the stack but it forces you to think about how those nested parts of your app will fail as you travel back up the stack.

[1]: https://doc.rust-lang.org/rust-by-example/std/result/questio...

Re: Cloudflare outage on November 18, 2025 post mortem

#513

Post by YetAnotherNick »

sznio wrote: post #471
jcalvinowens wrote: post #275

Earlier quoted context omitted.

> This is the multi-million dollar .unwrap() story. That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

>If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same. If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

In general for unexpected errors like these the internal function should log the error, and I assume it was either logged or they can quickly deduce reason based on the line number.

Re: Cloudflare outage on November 18, 2025 post mortem

#514

Post by NicoJuicy »

JB_Dev wrote: post #443
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Does their ring based rollout really truly have to be 0->100% in a few seconds? I don’t really buy this requirement. At least make it configurable with a more reasonable default for “routine” changes. E.g. ramping to 100% over 1 hour. As long as that ramp rate is configurable, you can retain the ability to respond fast to attacks by setting the ramp time to a few seconds if you truly think it’s needed in that moment.

I think defence against a DDOS against your network is the best reason for a quick rollout

Re: Cloudflare outage on November 18, 2025 post mortem

#516

Post by karel-3d »

richardwhiuk wrote: post #470
karel-3d wrote: post #468

Earlier quoted context omitted.

As a gopher I never understand why is there so many unwraps in an average rust code. Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

Because Go silently gives you zero/null instead

Idiomatically, it gives you `err` and you do `if err != nil {return err}`. While in rust you mostly do `.unwrap` and panic.

It's not in the type system, but it's idiomatic

Re: Cloudflare outage on November 18, 2025 post mortem

#517

Post by Buttons840 »

otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

When a failsafe system fails, it fails by failing to fail safely.

Re: Cloudflare outage on November 18, 2025 post mortem

#518

Post by smt88 »

Pooge wrote: post #498
smt88 wrote: post #392

Earlier quoted context omitted.

Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software. What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.

Costs to architect systems that serve millions of request daily have gone down . Not up. Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client. People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)

> Costs to architect systems that serve millions of request daily have gone down. Not up.

I never said serving millions of requests is more expensive. Protecting your servers is more expensive.

> Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.

HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloudflare's CDN/proxy is giving to HN incredibly cheaply, then running HN at the edge with good performance (and protecting it from botnets) would costs hundreds of millions of dollars.

> People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)

Again, attacking your own straw men here.

Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.

Post reply on HN