Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

661–670 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#661

Post by loglog »

never_inline wrote: post #377
bigstrat2003 wrote: post #315

Earlier quoted context omitted.

I'm with you! Checked exceptions are actually good and the hate for them is super short sighted. The exact same criticisms levied at checked exceptions apply to static typing in general, but people acknowledge the great value static types have for preventing errors at compile time. Checked exceptions have that same value, but are dunked on for some reason.

The dislike is probably because of 2 reasons. 1. in most cases they don't want to handle `InterruptedException` or `IOException` and yet need to bubble them up. In that case the code is very verbose. 2. it makes lambdas and functions incompatible. So eg: if you're passing a function to forEach, you're forced to wrap it in runtime exception. 3. Due to (1) and (2), most people become lazy and do `throws Exception` whic…

Java checked exceptions suffer from a lack of generic exception types ("throws T", where T can be e.g. "Exception", "Exception1|Exception2", or "never") This would also require union types and a bottom type. Without generics, higher order functions are very hard to use.

Re: Cloudflare outage on November 18, 2025 post mortem

#663

Post by janpio »

kqr wrote: post #643

One of the remediations listed is > Eliminating the ability for core dumps or other error reports to overwhelm system resources but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.

I understood this to be related to this section:

> As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.

(Just above https://blog.cloudflare.com/18-november-2025-outage/#how-clo...)

Re: Cloudflare outage on November 18, 2025 post mortem

#664

Post by echelon »

yakshaving_jgt wrote: post #566
echelon wrote: post #346

Earlier quoted context omitted.

Cargo needs to grow a label for crates that provably do not panic. (Neverminding allocations and things outside our control flow.) I want to ban crates that panic from my dependency chain. The language could really use an extra set of static guarantees around this. I would opt in.

This sounds a little bit like Safe Haskell , which never really took off.

I would be fine just getting rid of unwrap(), expect(), etc. That's still a net win.

Look at how many lazy cases of this there are in Rust code [1].

Some of these are no doubt tested (albeit impossible to statically guarantee), but a lot of it looks like sloppiness or not leaning on the language's strong error handling features.

It's disappointing to see. We've had so much of this creep into the language that eventually it caused a major stop-the-world outage. This is unlikely to be the last time we see it.

[1] https://github.com/search?q=unwrap%28%29+language%3ARust&typ...

Re: Cloudflare outage on November 18, 2025 post mortem

#665

Post by kosolam »

polack wrote: post #570
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

They failed on so many levels here. How can you write the proxy without handling the config containing more than the maximum features limit you set yourself? How can the database export query not have a limit set if there is a hard limit on number of features? Why do they do non-critical changes in production before testing in a stage environment? Why did they think this was a cyberattack and only after two hours rea…

So where are you migrating to?

Re: Cloudflare outage on November 18, 2025 post mortem

#667

Post by mattacular »

gwd wrote: post #561
brabel wrote: post #495

Earlier quoted context omitted.

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handl…

> Yes, I always thought it was wrong to use unwrap in examples. And because it gets picked up by LLMs. It would be interesting to know if this particular .unwrap() was written by a human.

There is a prevailing mentality that LLMs make it easy to become productive in new languages, if you are already proficient in one. That's perhaps true until you suddenly bump up against the need to go beyond your superficial understanding of the new language and its idiosyncrasies. These little collisions with reality occur until one of them sparks an issue of this magnitude.

In theory, experienced human code reviewers can course correct newer LLM-guided devs work before it blows up. In practice, reviewers are already stretched thin and submitters absolute to now rapidly generate more and more code to review makes that exhaustion effect way worse. It becomes less likely they spot something small but obvious amongst the haystack of LLM generated code bailing there way.

Re: Cloudflare outage on November 18, 2025 post mortem

#668

Post by mongol »

rkachowski wrote: post #579
abalone wrote: post #380

Earlier quoted context omitted.

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

Classic @devops_borat "To make error is human. To propagate error to all server in automatic way is #devops"

I miss him. It must be more than 10 years now

Re: Cloudflare outage on November 18, 2025 post mortem

#669

Post by yakshaving_jgt »

echelon wrote: post #664
yakshaving_jgt wrote: post #566

Earlier quoted context omitted.

This sounds a little bit like Safe Haskell , which never really took off.

I would be fine just getting rid of unwrap(), expect(), etc. That's still a net win. Look at how many lazy cases of this there are in Rust code [1]. Some of these are no doubt tested (albeit impossible to statically guarantee), but a lot of it looks like sloppiness or not leaning on the language's strong error handling features. It's disappointing to see. We've had so much of this creep into the language that eventua…

I don't write Rust so I don't really know, but from someone else's description here it sounds similar to `fromJust` in Haskell which is a common newbie footgun. I think you're right that this is a case of not using the language properly, though I know I was seduced into the idea that Haskell is safe by default when I was first learning, which isn't quite true — the safety features are opt-in.

A language DX feature I quite like is when dangerous things are labelled as such. IIRC, some examples of this are `accursedUnutterablePerformIO` in Haskell, and `DO_NOT_USE_OR_YOU_WILL_BE_FIRED_EXPERIMENTAL_CREATE_ROOT_CONTAINERS` in React.js.

Re: Cloudflare outage on November 18, 2025 post mortem

#670

Post by meltyness »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

tokio default behavior within a task is to ignore panics, such as an Err/None unwrap, and only crash that task, so it's impact limited so that's nice, maybe that's where the snowblindness came from.

it'd be kinda hard to amend the clippy lints to ignore coroutine unwraps but still pipe up on system ones. i guess.

edit: i think they'd have to be "solely-task-color-flavored" so definitely probably not trivial to infer

Post reply on HN