Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

101–110 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#101

Post by yoyohello13 »

arccy wrote: post #89
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

So… basically every language ever?

Except maybe Haskell.

Re: Cloudflare outage on November 18, 2025 post mortem

#102

Post by RagingCactus »

Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate.

Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as the downtime starts. And the well defined clean crashes should in theory also stand out against all the random errors that start occuring all over the system as it begins to go down, precisely because it's always failing at the exact same point.

Re: Cloudflare outage on November 18, 2025 post mortem

#103

Post by tptacek »

olivia-banks wrote: post #99
tptacek wrote: post #69

Earlier quoted context omitted.

What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.

What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than…

Yeah, see, that's what I mean.

Re: Cloudflare outage on November 18, 2025 post mortem

#104

Post by [deleted] »

Scaevolus wrote: post #97
otterley wrote: post #19

> work has already begun on how we will harden them against failures like this in the future. In particular we are: > Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input > Enabling more global kill switches for features > Eliminating the ability for core dumps or other error reports to overwhelm system resources > Reviewing failure modes for error conditio…

Global configuration is useful for low response times to attacks, but you need to have very good ways to know when a global config push is bad and to be able to rollback quickly. In this case, the older proxy's "fail-closed" categorization of bot activity was obviously better than the "fail-crash", but every global change needs to be carefully validated to have good characteristics here. Having a mapping of which ser…

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#105

Post by nine_k »

otterley wrote: post #96
arccy wrote: post #89

Earlier quoted context omitted.

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

https://en.wikipedia.org/wiki/Crash-only_software

Works when you have the Erlang system that does graceful handing for you: reporting, restarting.

Re: Cloudflare outage on November 18, 2025 post mortem

#107

Post by nickmonad »

koakuma-chan wrote: post #38
tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Why is there a 200 limit on appending names?

Limits in systems like these are generally good. They mention the reasoning around it explicitly. It just seems like the handling of that limit is what failed and was missed in review.

Re: Cloudflare outage on November 18, 2025 post mortem

#108

Post by [deleted] »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#109

Post by jijji »

this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about
Post reply on HN