Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

681–690 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#681

Post by [deleted] »

dehrmann wrote: post #454
never_inline wrote: post #377

Earlier quoted context omitted.

The dislike is probably because of 2 reasons. 1. in most cases they don't want to handle `InterruptedException` or `IOException` and yet need to bubble them up. In that case the code is very verbose. 2. it makes lambdas and functions incompatible. So eg: if you're passing a function to forEach, you're forced to wrap it in runtime exception. 3. Due to (1) and (2), most people become lazy and do `throws Exception` whic…

> 2. it makes lambdas and functions incompatible. This is true, but the hate predated lambdas in Java.

[deleted]

Re: Cloudflare outage on November 18, 2025 post mortem

#682

Post by DoctorOW »

jgilias wrote: post #658
vbezhenar wrote: post #607

Earlier quoted context omitted.

IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts). So basically bad config should be explicitly processed and handled by rolling back to…

You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.

> If that’s the case, the old config is already in memory when the new config is being parsed

I think that's explicitly a non-goal. My understanding is that Cloudflare prefers fail safe (blocking legitimate traffic) over fail open (allowing harmful traffic).

Re: Cloudflare outage on November 18, 2025 post mortem

#683

Post by antihero »

hedora wrote: post #186
vlovich123 wrote: post #162

Earlier quoted context omitted.

To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open). But more generally you could catch the panic at the FL2 layer to make that decision intentio…

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

The unwrap should be replaced by code that creates enough alerting to make a P0 incidident from their canary deployment immediately.

OR even, the bot code crashing should itself be generating alerts.

Canary deployment would be automatically rolled back until P0 incident resolved.

All of this could probably have happened and contained at their scale in less than a minute as they would likely generate enough "omg the proxy cannot handle its config" alerts off of a deployment of 0.001% near immediately.

Re: Cloudflare outage on November 18, 2025 post mortem

#684

Post by eastdakota »

SerCe wrote: post #9

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.

Re: Cloudflare outage on November 18, 2025 post mortem

#685

Post by quotemstr »

ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

If the error had been an exception instead of a result, could have bubbled up

I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled.

The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it.

Rust encourages developers to consider only the happy path. No wonder it's popular among people who've never had to deal with failure.

All of the concomitant complexity--- Result, ?, the test thing, anyhow, the inability for stdlib to report allocation failure --- is downstream of a fashion statement against exceptions Rust cargo-culted from Go.

The funniest part is that Rust does have exceptions. It just calls them panics. So Rust code has to deal with the ergonomic footgun of Result but pays anyway for the possibility of exceptions. (Sure, you can compile with panic=abort. You can't count on it.)

I could not be more certain that Rust should have been a language with exceptions, not Result, and that error objects are a gross antipattern we'll regret for decades.

Re: Cloudflare outage on November 18, 2025 post mortem

#686

Post by twoodfin »

abalone wrote: post #567
dev_l1x_be wrote: post #484

Earlier quoted context omitted.

Exactly the right take. Even when you want to have rapid changes on your infra, do it at least by region. You can start with the region where the least amount of users are impacted and if everything is fine, there is no elevated number of crashes for example, you can move forward. It was a standard practice at $RANDOM_FAANG when we had such deployments.

Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.

But the rapid deployment mechanism for bot features wasn’t where the bug was introduced.

In fact, the root bug (faulty assumption?) was in one or more SQL catalog queries that were presumably written some time ago.

(Interestingly the analysis doesn’t go into how these erroneous queries made it into production OR whether the assumption was “to spec” and it’s the security principal change work that was faulty. Seems more likely to be the former.)

Re: Cloudflare outage on November 18, 2025 post mortem

#687

Post by kevincox »

watchful_moose wrote: post #584
hnarn wrote: post #508

> That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. > The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That…

Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.

I'm shocked that an automatic canary rollout wasn't an action item. Pushing anything out globally is a guaranteed failure again in the future.

Even if you want this data to be very fresh you can probably afford to do something like:

1. Push out data to a single location or some subset of servers.

2. Confirm that the data is loaded.

3. Wait to observe any issues. (Even a minute is probably enough to catch the most severe issues.)

4. Roll out globally.

Re: Cloudflare outage on November 18, 2025 post mortem

#688

Post by seethishat »

abalone wrote: post #380
ojosilva wrote: post #42

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is text…

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is: 1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually. 2. Despite the elevated risk of system…

The bot is efficient. This is by design. It will push out mistakes just as efficiently as it pushes out good changes. Good or bad... the plane of control is unchanged.

This is the danger of automated control systems. If they get hacked or somehow push out bad things (CloudStrike), they will have complete control and be very efficient.

Re: Cloudflare outage on November 18, 2025 post mortem

#689

Post by eastdakota »

ynx0 wrote: post #314
eastdakota wrote: post #95

Earlier quoted context omitted.

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current…

How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups. Thanks for the insight.

Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.

Re: Cloudflare outage on November 18, 2025 post mortem

#690

Post by echelon »

abigailphoebe wrote: post #638
echelon wrote: post #297

Earlier quoted context omitted.

No. Don't say "you're holding it wrong". The language says "safe" on the tin. It advertises safety. This shouldn't be possible. This is a null pointer. In Rust. Unwrap needs to die. We should all fight to remove it.

you either misunderstand the rust ethos or are intentionally misrepresenting it. safe refers to memory safety. once again, if you write bad code, that’s your fault, not the languages. this is a feature of rust that was used incorrectly.

Rust has grown beyond its original design as a "memory safe" language. People are using this as an HTTP/RPC server programming language now. WASM serverless jobs, etc. Rust has found itself deployed in a lot of unexpected places.

These folks are not choosing Rust for the memory safety guarantees. They're choosing Rust for being a fast language with a nice type system that produces "safe" code.

Rust is widely known for producing relatively defect-free code on account of its strong type system and ergonomics. Safety beyond memory safety.

Unwrap(), expect(), and their kin are a direct affront to this.

There are only two uses cases for these: (1) developer laziness, (2) the engineer spent time proving the method couldn't fail, but unfortunately they're not using language design features that allow this to be represented in the AST with static guarantees.

In both of these cases, the engineer should instead choose to (1) pass the Result or Option to the caller and let the caller decide what to do, (2) do the same, but change the type to be more appropriate to the caller, (3) handle it locally so the caller doesn't have to deal with it, (4) silently turn it into a success. That's it. That's idiomatic Rust.

This should be concerning to everyone:

https://github.com/search?q=unwrap%28%29+language%3ARust&typ...

I'm now panicked (hah) that some dependency of mine will unwrap something and panic at runtime. That's entirely invisible to users. It's extremely dangerous.

Today a billion people saw the result of this laziness. It won't be the last time. And hopefully it never happens in safety-critical applications like aircraft. But the language has no say in this because it isn't taking a stand against this unreasonably sharp edge yet. Hopefully it will. It's a (relatively) easy fix.

Post reply on HN