Post Mortem on Cloudflare Control Plane and Analytics Outage

Of the incident? Someone on my team called me about 30 minutes after it started. It was challenging for me to stay on top of because it was also the same day as our Q3 earnings call. But team kept me informed throughout the day. I helped where I could. And they handled a very difficult situation very well. That said, lots we can learn from and improve.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#5

Post by divbzero » Sat, Nov 04, 2023, 6:49 AM UTC

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would cause all three to be impacted, while still close enough that they could all run active-active redundant data clusters.

If the three data centers are all around Hillsboro, Oregon, an earthquake could probably take out all three simultaneously.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#6

Post by tebbers » Sat, Nov 04, 2023, 6:51 AM UTC

I think overall Cloudflare did a decent job on this. Clearly the DC provider cocked up big time here, but Cloudflare kept running fine for the vast majority of customers globally. No system is perfect and it’s only apocalyptic scenarios like this where the vulnerabilities are exposed - and they will now be fixed. Hope the SRE guys got some rest after all that stress.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#7

Post by austinkhale » Sat, Nov 04, 2023, 6:52 AM UTC

I love how thorough Cloudflare post mortem’s are. Reading the frank, transparent explanations are like a breath of fresh air compared to the obfuscation of nearly every other company comm’s strategy.

We were affected but it’s blog posts like these that make me never want to move away. Everyone makes mistakes. Everyone has bad days. It’s how you react afterwards that makes the difference.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#8

Post by nanankcornering » Sat, Nov 04, 2023, 6:57 AM UTC

divbzero wrote: post #5

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would caus…

thats why DRC exists right?

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#9

Post by [deleted] » Sat, Nov 04, 2023, 7:02 AM UTC

[deleted]

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#10

Post by TekMol » Sat, Nov 04, 2023, 7:13 AM UTC

Contrary to others here, I find the postmortem a bit lacking.

The TLDR is that CF runs in multiple data centers, one went down, and the services that depend on it went down with it.

The interesting question would be why those services did depend on a single data center.

They are pretty vague about it

    Cloudflare allows multiple teams to innovate quickly. As such,
    products often take different paths toward their initial alpha.

If I was the CEO, I would look into the specific decisions of the engineers and why they decided to make services depend on just one data center. That would make an interesting blog post to me.

Designing a highly available system and building a company fast leads to interesting tradeoffs. The details would be interesting.