Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

651–660 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#651

Post by echelon »

bigstrat2003 wrote: post #347
echelon wrote: post #327

Earlier quoted context omitted.

Panics should be explicit, not implicit. unwrap() should effectively work as a Result where the user must manually invoke a panic in the failure branch. Make special syntax if a match and panic is too much boilerplate. This is like an implicit null pointer exception that cannot be statically guarded against. I want a way to statically block any crates doing this from my dependency chain.

unwrap is explicit.

What happens if a dependency of mine unwrap()s or expect()s something that is an error or isn't there? What should I do?

How was I informed as a user? It's not in the type signature.

Sounds like I get to indeterminately crash at runtime and have a fun time debugging.

Re: Cloudflare outage on November 18, 2025 post mortem

#652

Post by peanut-walrus »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I wonder if similar to infrastructure resilience, code resilience is also required for critical services that can never go down? Instead of relying on a single implementation for a critical service, have multiple independent implementations in different languages. Back when I was running my own DNS servers, I did always ensure that primary and secondary were running on different platforms and different software.

Re: Cloudflare outage on November 18, 2025 post mortem

#653

Post by polack »

raxxorraxor wrote: post #575
polack wrote: post #570

Earlier quoted context omitted.

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors. But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.

What is not realistic? To do simple input validation on data that has the potential to break 20% of the internet? To not have a system in place to rollback to the latest known state when things crash?

Cloudflare builds a global scale system, not an iphone app. Please act like it.

Re: Cloudflare outage on November 18, 2025 post mortem

#654

Post by VMG »

csomar wrote: post #501
ozgrakkurt wrote: post #459

Earlier quoted context omitted.

And the error magically disappears when the function returns it?

It doesn’t disappear, it forces you to handle it.

Propagating upwards a valid way of handling it and often the correct answer.

There needs to be something at the top level that can handle a crashing process.

Re: Cloudflare outage on November 18, 2025 post mortem

#655

Post by echelon »

milliams wrote: post #600
tormeh wrote: post #525

Earlier quoted context omitted.

Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code. And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.

[lints.clippy] dbg_macro = "deny" unwrap_used = "deny" expect_used = "deny"

https://github.com/search?q=unwrap%28%29+language%3ARust&typ...

This is sobering.

My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness.

Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.

Re: Cloudflare outage on November 18, 2025 post mortem

#656

Post by discordianfish »

bri3d wrote: post #260

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random…

Indeed, nothing about the root issues are particular surprising but why they missed a critical service panicing across their fleet is not bubbling up.

My best guess is too many alerts firing without a clear hierarchy and possibilities to seprate cause from effect. It's a typical challenge but I wish they would shed some light on that. And its a bit concerning that improving observability is not part of their follow up steps.

Re: Cloudflare outage on November 18, 2025 post mortem

#657

Post by monerozcash »

purple_turtle wrote: post #608
Carriethebest wrote: post #587

Earlier quoted context omitted.

There are many self-hosted alternatives to protect against botnet. We don't have to use cloudflare. Everthing is under their control!

Can you name three of this many alternatives? How they magically manage DDOS larger than their bandwidth? If the plan is to have larger bandwidth than any DDOS it is going to be expensive, quickly.

You could probably get a very fat pipe with usage based billing, you'd only go bankrupt when you get hit by a big DDoS and not before.

Re: Cloudflare outage on November 18, 2025 post mortem

#658

Post by jgilias »

vbezhenar wrote: post #607
thatoneengineer wrote: post #413

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen". The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they w…

IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts). So basically bad config should be explicitly processed and handled by rolling back to…

You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.

Re: Cloudflare outage on November 18, 2025 post mortem

#659

Post by tatersolid »

Aeolun wrote: post #90
mewpmewp2 wrote: post #62

Earlier quoted context omitted.

But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day. In this case it seems the database table in question seemed modest in size (the features for ML) so naive…

That just means it takes longer to test. It may not be possible to do it in a reasonable timeframe with the volumes involved, but if you already have 100k servers running to serve 25M requests per second, maybe briefly booting up another 100k isn’t going to be the end of the world? Either way, you don’t need to do it on every commit, just often enough that you catch these kinds of issues before they go to prod.

> maybe briefly booting up another 100k isn’t going to be the end of the world

Cloudflare doesn’t run in AWS. They are a cloud provider themselves and mostly run on bare metal. Where would these extra 100k physical servers come from?

Re: Cloudflare outage on November 18, 2025 post mortem

#660

Post by echelon »

wrs wrote: post #350
dist1ll wrote: post #287

Earlier quoted context omitted.

Sure, these days I'm mostly working on a few compilers. Let's say I want to make a fixed-size SSA IR. Each instruction has an opcode and two operands (which are essentially pointers to other instructions). The IR is populated in one phase, and then lowered in the next. During lowering I run a few peephole and code motion optimizations on the IR, and then do regalloc + asm codegen. During that pass the IR is mutated a…

And it's fine for a compiler to panic when it violates an assumption. Not so with the Cloudflare code under discussion.

Idiomatic Rust would have been to return a Result to the caller, not to surprise them with a panic.

The developer was lazy.

A lot of Rust developers are: https://github.com/search?q=unwrap%28%29+language%3ARust&typ...

Post reply on HN