Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

831–840 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#831

Post by yoyohello13 »

samdoesnothing wrote: post #489
Patryk27 wrote: post #486

Earlier quoted context omitted.

> [...] it catches logic errors like this but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens > You won't see Go or Java developers making such strong claims about their preferred languages. yess no Java developer ever said that OOP will solve world hunger

> but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens The issue is that it wasn't fine to panic, thus Rust did not catch this error.

Come on now. You can't blame the compiler when the programmer explicitly told the compiler to not worry about it. There is nothing in existence that can protect against something like that.

Re: Cloudflare outage on November 18, 2025 post mortem

#832

Post by wrs »

mike_hearn wrote: post #760
bombela wrote: post #490

Earlier quoted context omitted.

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted. We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours. The most useful thing exceptions give you is not static compile time checking, it's the sta…

That has nothing to do with exceptions, just the ability to unwind the stack. Rust can certainly give you a backtrace on panics; you don’t even have to write a handler to get it. I would find it hard to believe Cloudflare’s services aren’t configured to do it. I suspect they just didn’t put the entire message in the post.

Re: Cloudflare outage on November 18, 2025 post mortem

#833

Post by wrs »

echelon wrote: post #651
bigstrat2003 wrote: post #347

Earlier quoted context omitted.

unwrap is explicit.

What happens if a dependency of mine unwrap()s or expect()s something that is an error or isn't there? What should I do? How was I informed as a user? It's not in the type signature. Sounds like I get to indeterminately crash at runtime and have a fun time debugging.

Same thing that would happen if it did a match statement and panicked. The problem is the panic, not the unwrap.

I don’t think you can ever completely eliminate panics, because there are always going to be some assumptions in code that will be surprisingly violated, because bugs exist. What if the heap allocator discovers the heap is corrupted? What if you reference memory that’s paged out and the disk is offline? (That one’s probably not turned into a panic, but it’s the same principle.)

Re: Cloudflare outage on November 18, 2025 post mortem

#834

Post by wrs »

littlestymaar wrote: post #648
wrs wrote: post #418

Earlier quoted context omitted.

I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.

I don't understand your point: panic! is akin to expect : you think about it consciously, use it explicitly and you write down a panic message explaining its rational. unwrap isn't like that.

It should be. If you aren’t treating it exactly the same as panic and expect, that’s what I’m calling the “blind spot”. And why should you have to make up a message every time when the backtrace is going to tell you what was wrong?

Re: Cloudflare outage on November 18, 2025 post mortem

#835

Post by Matthias247 »

When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.

The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".

The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.

Re: Cloudflare outage on November 18, 2025 post mortem

#836

Post by kevin_thibedeau »

an_ko wrote: post #698
echelon wrote: post #655

Earlier quoted context omitted.

https://github.com/search?q=unwrap%28%29+language%3ARust&typ... This is sobering. My new fear is some dependency unwrap()ing or expect()ing something where they didn't prove the correctness. Unwrap() and expect() are an anti-pattern and have no place in idiomatic Rust code. The language should move to deprecate them.

I use unwrap a lot, and my most frequent target is unwrapping the result of Mutex::lock. Most applications have no reasonable way to recover from lock poisoning, so if I were forced to write a match for each such use site to handle the error case, the handler would have no choice but to just call panic anyway. Which is equivalent to unwrap, but much more verbose. Perhaps it needs a scarier name, like "assume_ok".

> the handler would have no choice but to just call panic anyway

The handler could log the error and then panic. Much better than chasing bad hunches about a DDoS.

Re: Cloudflare outage on November 18, 2025 post mortem

#837

Post by nijave »

matteocontrini wrote: post #588
NicoJuicy wrote: post #514

Earlier quoted context omitted.

I think defence against a DDOS against your network is the best reason for a quick rollout

This was not about DDoS defense but the Bot Management feature, which is a paid Enterprise-only feature not enabled by default to block automated requests regardless of whether an attack is going on. https://developers.cloudflare.com/bots/get-started/bot-manag...

Bots can also cause a DoS/DDoS. We use the feature to restrict certain AI scraper tools by user agent that adversly impact performance (they have a tendency to hammer "export all the data" endpoints much more than regular users do)

Re: Cloudflare outage on November 18, 2025 post mortem

#838

Post by ogurechny »

> Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable.

Excuse me, what you've just said? Who decided on “Cloudflare's importance in the Internet ecosystem”? Some see it differently, you know, there's no need for that self-assured arrogance of an inseminating alpha male.

Re: Cloudflare outage on November 18, 2025 post mortem

#839

Post by arifiqbal81 »

Thanks for the detailed writeup and explaining the root cause in details

However, I have a question from a release deployment process perspective. Why was this issue not detected during internal testing ? I didn't find the RCA analysis covering this aspect. Doesn't cloudflare have an internal test stage as part of its CICD pipeline. Looking the description of the issue, it should have been immediately detected in internal stage test environment.

Re: Cloudflare outage on November 18, 2025 post mortem

#840

Post by [deleted] »

gizmo686 wrote: post #363
forsalebypwner wrote: post #150

Earlier quoted context omitted.

I think the reasoning behind this is because of the nature of the file being pushed - from the post mortem: "This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change thei…

In this case, the file fails quickly. A pretest that consists of just attempting to load the file would have caught it. Minutes is more than enough time to perform such a check.

[deleted]
Post reply on HN