Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

221–230 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#221

Post by butvacuum »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

It rang more as "A/B deployments are pointless if you can't tell if a downstream failure is related." To me.

Re: Cloudflare outage on November 18, 2025 post mortem

#222

Post by notatoad »

verletzen wrote: post #85
notatoad wrote: post #55

Earlier quoted context omitted.

it seems like a good chance that despite thinking their status page was completely independent of cloudfront, enough of the internet is dependent on cloudfront now that they're simply wrong about the status page's independence.

i think you've got cloudflare and cloudfront mixed up.

ahah oops. yeah, it's a problem. i've got two projects ongoing that each rely on one of them, and i can never keep it straight.

Re: Cloudflare outage on November 18, 2025 post mortem

#224

Post by pdimitar »

duped wrote: post #218
pdimitar wrote: post #208

While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language. They are escape hatches. Without those your language would never take off. But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to…

This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable. Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do. If panicking is guaranteed because of some input mistake to the system your failure is in testing.

I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit.

I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all.

And you will not believe how many alerts I got for the "impossible" errors.

I do agree there was not too much that could have been done, yes. But they should have invested in more visibility and be more thorough. I mean, hobbyist Rust devs seem to do that better.

It was just a bit disappointing for me. As mentioned above, I'd understand and sympathise with many other mistakes but this one stung a bit.

Re: Cloudflare outage on November 18, 2025 post mortem

#225

Post by dist1ll »

10000truths wrote: post #189
dist1ll wrote: post #149

Earlier quoted context omitted.

> every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this. How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.

Yes? Funnily enough, I don't often use indexed access in Rust. Either I'm looping over elements of a data structure (in which case I use iterators), or I'm using an untrusted index value (in which case I explicitly handle the error case). In the rare case where I'm using an index value that I can guarantee is never invalid (e.g. graph traversal where the indices are never exposed outside the scope of the traversal),…

If that's the case then hats off. What you're describing is definitely not what I've seen in practice. In fact, I don't think I've ever seen a crate or production codebase that documents infallibility of every single slice access. Even security-critical cryptography crates that passed audits don't do that. Personally, I found it quite hard to avoid indexing for graph-heavy code, so I'm always on the lookout for interesting ways to enforce access safety. If you have some code to share that would be very interesting.

Re: Cloudflare outage on November 18, 2025 post mortem

#226

Post by HumanOstrich »

otterley wrote: post #180
tptacek wrote: post #167

Earlier quoted context omitted.

I recoil from that last statement not because I have a rooting interest in Cloudflare but because the last several years of working at Fly.io have drilled Richard Cook's "How Complex Systems Fail"† deep into my brain, and what you said runs aground of Cook #18: Failure free operations require experience with failure. If the exact same thing happens again at Cloudflare, they'll be fair game. But right now I feel peopl…

Suppose they did have the cellular architecture today, but every other fact was identical. They'd still have suffered the failure! But it would have been contained , and the damage would have been far less. Fires happen every day. Smoke alarms go off, firefighters get called in, incident response is exercised, and lessons from the situation are learned (with resulting updates to the fire and building codes). Yet even…

What variant of cellular architecture are you referring to? Can you give me a link or few? I'm fascinated by it and I've led a team to break up a monolithic solution running on AWS to a cellular architecture. The results were good, but not magic. The process of learning from failures did not stop, but it did change (for the better).

No matter what architecture, processes, software, frameworks, and systems you use, or how exhaustively you plan and test for every failure mode, you cannot 100% predict every scenario and claim "cellular architecture fixes this". This includes making 100% of all failures "contained". Not realistic.

Re: Cloudflare outage on November 18, 2025 post mortem

#229

Post by rawgabbit »

rawgabbit wrote: post #7

> The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database: SELECT name, type FROM system.columns WHERE table = 'http_requests_features' order by name; Note how the query does not filter for the database name. Wit…

Here is a bit more context in addition to the quote above. A ClickHouse permissions change made a metadata query start returning duplicate column metadata from an extra schema, which more than doubled the size and feature count of a Bot Management configuration file. When this oversized feature file was deployed to edge proxies, it exceeded a 200-feature limit in the bot module, causing that module to panic and the core proxy to return 5xx errors globally

Re: Cloudflare outage on November 18, 2025 post mortem

#230

Post by snoppy45 »

I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.
Post reply on HN