Post Mortem on Cloudflare Control Plane and Analytics Outage • Hacker News Board

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#161

Post by arrakeenrevived » Sat, Nov 04, 2023, 4:35 PM UTC

marcinzm wrote: post #83

creshal wrote: post #48

Earlier quoted context omitted.

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment! > We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to inn…

> Complete and utter management failure. And customers apparently are sold what Cloudflare internally considers to be alpha quality software? This has been my experience with AWS and GCP as well. Assume anything that's under 3 years old is not really GA quality no matter what they say publicly.

I've been involved with some new service launches at AWS, and it's a strict requirement that everything goes through some rigorous operational and security reviews that cover exactly these issues before the service can be launched as GA. Feature-wise people might consider them "alpha", but when it comes to the resilience and security of the launched features, they are held to much higher standards than what is being described in this post-mortem.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#162

Post by arrakeenrevived » Sat, Nov 04, 2023, 4:44 PM UTC

jmbwell wrote: post #135

ShadowRegent wrote: post #74

Interesting choice to spend the bulk of the article publicly shifting blame to a vendor by name and speculating on their root cause. Also an interesting choice to publicly call out that you're a whale in the facility and include an electrical diagram clearly marked Confidential by your vendor in the postmortem. Honestly, this is rather unprofessional. I understand and support explaining what triggered the event and g…

If Flexential and PGE aren't sharing information or otherwise cooperating as much as Cloudflare might like, then going public with some speculation might be an attempt at applying some pressure to get to the bottom of what happened. It might also be an effort to get out in front of the story before someone else does the speculating. In any case, with at least three parties involved, with multiple interconnected syste…

>If Flexential and PGE aren't sharing information or otherwise cooperating as much as Cloudflare might like, then going public with some speculation might be an attempt at applying some pressure to get to the bottom of what happened.

It's been 2 days. I doubt PGE or Flexential even have root caused it yet, and even if they have, good communication takes time.

You don't throw someone under the bus and smear their name publicly just because they haven't replied for two days, and you certainly don't start speculating on their behalf. That's bad partnership.

You also don't publicly share what "Flexential employees shared with us unofficially" (quote from the article) - what a great way to burn trust with people who probably told you stuff in confidence.

>if Cloudflare is going to effectively anticipate this cluster of failure modes in future design decisions, it's reasonable for them to want to know what happened all the way down.

They can do all of that without smearing people on their company blog. In fact, they can do all of that without even knowing what happened to PGE/Flexential, because per their own admission they were already supposed to be anticipating this, but failed at it. Power outages and data center issues are a known thing, and is exactly why HA exists. HA which Cloudflare failed at. This post-mortem should be almost entirely about that failure rather than speculation about a power outage.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#163

Post by organsnyder » Sat, Nov 04, 2023, 4:50 PM UTC

creshal wrote: post #48

Dunedan wrote: post #21

> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discover…

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment! > We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to inn…

Having worked at companies with varying degrees of autonomy, in my experience a more flexible structure allows for building systems that are ultimately more resilient. Of course, there are ways to do it poorly, but that doesn’t mean it’s a “complete and utter management failure”.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#164

Post by stemlord » Sat, Nov 04, 2023, 4:52 PM UTC

I'm ultimately glad this happened because it very effectively helps illustrate how we are assigning a centralized gatekeeper to the internet at the infrastructure level and why it's a bad thing.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#165

Post by jsnell » Sat, Nov 04, 2023, 4:54 PM UTC

andyjohnson0 wrote: post #66

> We are a relatively large customer of the facility, consuming approximately 10 percent of its total capacity. I'm surprised that CF are renting space in colocation facilities. I would have expected a business of their size to have their own DCs. Is this common practice for cloud providers?

The part you're missing is that their business is not actually that large (<$1 billion of revenue in 2022 - still deep in the red - and 3k employees).

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#166

Post by simple10 » Sat, Nov 04, 2023, 5:22 PM UTC

This is good reminder to myself to transfer domains registered on Cloudflare to another provider and only use Cloudflare for DNS or vice versa. I was effectively locked out of making any changes to domains registered and DNS hosted on Cloudflare during the entire outage due to single point of failure (Cloudflare) on my part.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#167

Post by fanf2 » Sat, Nov 04, 2023, 5:28 PM UTC

pests wrote: post #11

divbzero wrote: post #5

Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would caus…

> Hillsboro, Oregon, an earthquake could probably take out all three simultaneously. Is it west of I5? (yes) Oh yeah, they all gone. Cascadia Subduction Zone - https://pnsn.org/outreach/earthquakesources/csz

There’s also the Portland Hills and Gales Creek fault zones to worry about.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#168

Post by NicoJuicy » Sat, Nov 04, 2023, 5:32 PM UTC

yowai wrote: post #155

NicoJuicy wrote: post #98

Earlier quoted context omitted.

Everything was on the status page since the start? DC related updates: > Update - Power to Cloudflare’s core North America data center has been partially restored. Cloudflare has failed over some core services to a backup data center, which has partially remediated impact. Cloudflare is currently working to restore the remaining affected services and bring the core North America data center back online. Nov 02, 2023…

As an enterprise customer, I would expect a CSM reaching out to us informing us about the impact, getting into more details about any restoration plans and potentially even ETAs or rough prioritization to resolution on them. In reality, Cloudflare's support team was essentially completely unavailable on Nov 2, leaving only the status page. And for most of the day, the updates on the status page were very sparse excep…

?

1) Were you affected on the data plane? Which product?

As far as I can tell, while the outage was in the core dc's. The impact was minor.

2) Both examples were exactly from 2 November. Not 3 November.

3) What method of support did you try? I thought that their support was impacted ( email?).

The status page explicitly mentioned to get in contact with your account manager for some config changes on some products, if you wanted changes.

4) I have never heard of Enterprise customers being contacted by a cloud company during an outage.

Which company does that? Do you have an example?

5) I would think it's absolutely a nogo to contact every preemptively Enterprise customer with: "hey, the product works, but if you change xyz, atm that doesn't.".

Since most customers weren't affected and some others were minorly impacted.

There is not a single cloud company that does that.

Feel free to correct me if I'm wrong...

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#169

Post by cortesoft » Sat, Nov 04, 2023, 5:33 PM UTC

andyjohnson0 wrote: post #66

> We are a relatively large customer of the facility, consuming approximately 10 percent of its total capacity. I'm surprised that CF are renting space in colocation facilities. I would have expected a business of their size to have their own DCs. Is this common practice for cloud providers?

CF is probably a lot smaller than you realize, especially per data center. As someone who works at a different CDN, I am guessing they only have a few hundred machines per data center around the world. That is way too small to be able to run your own DC.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#170

Post by alphabettsy » Sat, Nov 04, 2023, 5:39 PM UTC

arrakeenrevived wrote: post #162

jmbwell wrote: post #135

Earlier quoted context omitted.

If Flexential and PGE aren't sharing information or otherwise cooperating as much as Cloudflare might like, then going public with some speculation might be an attempt at applying some pressure to get to the bottom of what happened. It might also be an effort to get out in front of the story before someone else does the speculating. In any case, with at least three parties involved, with multiple interconnected syste…

>If Flexential and PGE aren't sharing information or otherwise cooperating as much as Cloudflare might like, then going public with some speculation might be an attempt at applying some pressure to get to the bottom of what happened. It's been 2 days. I doubt PGE or Flexential even have root caused it yet, and even if they have, good communication takes time. You don't throw someone under the bus and smear their name…

> You don't throw someone under the bus and smear their name publicly just because they haven't replied for two days, and you certainly don't start speculating on their behalf. That's bad partnership.

1. When you’re paying them the kind of money I imagine they’re paying and they don’t reply for 2 days, yea that’s crazy if true. I’d expect a client of this size could take to an executive on their personal number.

2. Telling the facts as you know them to be especially regarding very poor communication isn’t a smear.