Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare outage on November 18, 2025 post mortem

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 953 posts

Re: Cloudflare outage on November 18, 2025 post mortem

#31

Post by stefan_ »

tptacek wrote: post #21
tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Probably because this case was something more akin to an assert than an error check.

You are saying this would not have happened in a C release build where asserts define to nothing?

Wonder why these old grey beards chose to go with that.

Re: Cloudflare outage on November 18, 2025 post mortem

#32

Post by eastdakota »

lukan wrote: post #20

"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our stat…

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

Re: Cloudflare outage on November 18, 2025 post mortem

#33

Post by Aeolun »

EvanAnderson wrote: post #14

It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it. The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but t…

I’m fairly certain it will be after they read this thread. It doesn’t feel like they don’t want, or are incapable of improving?

Re: Cloudflare outage on November 18, 2025 post mortem

#34

Post by tptacek »

stefan_ wrote: post #31
tptacek wrote: post #21

Earlier quoted context omitted.

Probably because this case was something more akin to an assert than an error check.

You are saying this would not have happened in a C release build where asserts define to nothing? Wonder why these old grey beards chose to go with that.

I am one of those old grey beards (or at least, I got started shipping C code in the 1990s), and I'd leave asserts in prod serverside code given the choice; better that than a totally unpredictable error path.

Re: Cloudflare outage on November 18, 2025 post mortem

#35

Post by eastdakota »

gucci-on-fleek wrote: post #13

> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

Re: Cloudflare outage on November 18, 2025 post mortem

#37

Post by eastdakota »

tptacek wrote: post #17
EvanAnderson wrote: post #14

It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it. The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but t…

I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).

That’s correct.

Re: Cloudflare outage on November 18, 2025 post mortem

#38

Post by koakuma-chan »

tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Why is there a 200 limit on appending names?

Re: Cloudflare outage on November 18, 2025 post mortem

#39

Post by tptacek »

eastdakota wrote: post #37
tptacek wrote: post #17

Earlier quoted context omitted.

I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).

That’s correct.

Is it actually consul-template? (I have post-consul-template stress disorder).

Re: Cloudflare outage on November 18, 2025 post mortem

#40

Post by orphea »

sayrer wrote: post #18
tristan-morris wrote: post #16

Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

Yes, can't have .unwrap() in production code (it's ok in tests)

Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.
Post reply on HN