Post Mortem on Cloudflare Control Plane and Analytics Outage • Hacker News Board

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#181

Post by phan » Sat, Nov 04, 2023, 6:43 PM UTC

arrakeenrevived wrote: post #161

marcinzm wrote: post #83

Earlier quoted context omitted.

> Complete and utter management failure. And customers apparently are sold what Cloudflare internally considers to be alpha quality software? This has been my experience with AWS and GCP as well. Assume anything that's under 3 years old is not really GA quality no matter what they say publicly.

I've been involved with some new service launches at AWS, and it's a strict requirement that everything goes through some rigorous operational and security reviews that cover exactly these issues before the service can be launched as GA. Feature-wise people might consider them "alpha", but when it comes to the resilience and security of the launched features, they are held to much higher standards than what is being…

Your operational reviews must be lacking at AWS then (surprise surprise) then because there are so many instances where something will be released in alpha yet the documentation will still be outdated, stale and incorrect LOL.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#182

Post by phan » Sat, Nov 04, 2023, 6:45 PM UTC

arrakeenrevived wrote: post #175

alphabettsy wrote: post #170

Earlier quoted context omitted.

> You don't throw someone under the bus and smear their name publicly just because they haven't replied for two days, and you certainly don't start speculating on their behalf. That's bad partnership. 1. When you’re paying them the kind of money I imagine they’re paying and they don’t reply for 2 days, yea that’s crazy if true. I’d expect a client of this size could take to an executive on their personal number. 2. T…

They aren't telling the facts as they know them. Cloudflare themselves say that the information in the article is "speculation" (the article literally uses that term). Publicly casting blame based on speculation isn't something you do to someone that you want to have a good working relationship with, no matter how much money you pay them.

If you actually worked with datacenters you'd understand that what PGE and Flexential is unacceptable as well

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#183

Post by isbvhodnvemrwvn » Sat, Nov 04, 2023, 6:48 PM UTC

ecs78 wrote: post #150

sailingparrot wrote: post #132

Earlier quoted context omitted.

> Everyone makes mistakes. Everyone has bad days. The issue is when you start having bad days every other day though. We use and depend on CloudFlare Images heavily, it has now been down more than 67 hours over the last 30 days (22h on October 9th, 42h Nov 2 - Nov 4 and a sprinkle of ~hour long outages in between). That's 90.6% availability over the last month. Transparency is a great differentiator between providers…

They are a younger company than these other providers. Microsoft, Google, and AWS had their own growth pains and disasters. Remember when Microsoft deleted all the data (contacts, photos, etc) off all their customers Danger phones by accident and had no backup. Talk about naming their product a self-fulfilling prophecy.

Cloudflare is 14 years old and Cloudflare Stream, the "newer services they didn't have time to make HA" is 6 years old today.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#184

Post by mananaysiempre » Sat, Nov 04, 2023, 6:53 PM UTC

weird-eye-issue wrote: post #142

Wow they REALLY buried this important part didn't they! This took a ton of scrolling: "Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04." Bingo, there we have it.

There’s also the part where the disaster recovery site apparently fell over under the load (which, OK, is a thing that might happen) and they needed to code up limits on the fly (and that is not OK; I don’t have the slightest idea how one might test this, but if you’re building a “disaster” site it seems like you’d need to figure it out):

> When services were turned up there, we experienced a thundering herd problem where the API calls that had been failing overwhelmed our services. We implemented rate limits to get the request volume under control.

This seems not to be mentioned in the bullet points at the end of the text (which are otherwise reasonable).

And now I’m curious—how do you design cold failover when the system is complex enough to be metastable[1] and you can’t afford to test it on live traffic? I can guess which techniques you could use to build it, it’s the design and testing part (knowing the techniques actually work in your situation) that’s the problem.

One other thing that seems to have gone completely unmentioned:

> Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. [... W]e made the call at 13:40 UTC to fail over to Cloudflare's disaster recovery sites located in Europe.

Why did the decision take so long? I can imagine it can’t be made lightly, but two hours seems like too much hesitation, even if there was an expectation that power would be restored imminently for most of that time. There has to be a (predetermined?) point when you hit the switch regardless of any promises. Was it really set that far?

[1] http://charap.co/metastable-failures-in-distributed-systems/

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#185

Post by AzzyHN » Sat, Nov 04, 2023, 6:57 PM UTC

To preface: I am not qualified to talk about this in the slightest.

> Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04

What do you mean you discovered? How could you not know? Surely when you were setting this high availability cluster up years ago and migrating services over, you double checked that all crucial dependencies had also been moved, right? And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right???

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#186

Post by arrakeenrevived » Sat, Nov 04, 2023, 7:04 PM UTC

phan wrote: post #181

arrakeenrevived wrote: post #161

Earlier quoted context omitted.

I've been involved with some new service launches at AWS, and it's a strict requirement that everything goes through some rigorous operational and security reviews that cover exactly these issues before the service can be launched as GA. Feature-wise people might consider them "alpha", but when it comes to the resilience and security of the launched features, they are held to much higher standards than what is being…

Your operational reviews must be lacking at AWS then (surprise surprise) then because there are so many instances where something will be released in alpha yet the documentation will still be outdated, stale and incorrect LOL.

I think you misunderstand what's being talked about in this thread. "Operations" in this context has nothing to do with external-facing documentation, and instead refers to the resilience of the service and ensuring it doesn't for example, stop working when a single data center experiences a power outage.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#187

Post by ImPostingOnHN » Sat, Nov 04, 2023, 7:04 PM UTC

ShadowRegent wrote: post #128

ImPostingOnHN wrote: post #85

Earlier quoted context omitted.

I prefer to have their informed speculation here. Has Flexential provided a similarly detailed, public root cause analysis? If so, maybe we can refer to it. If not, how do you expect us to read it?

It’s only been a couple of business days, and it’s likely that they themselves will need root cause from equipment vendors (and perhaps information from the utility) to fully explain what happened. Perhaps they won’t publish anything, but at least give them an opportunity before trying to do it for them.

I expect them to start reporting out what they know immediately, and update as they learn more. If they're not doing that, and indeed haven't reported anything in days, that is a huge failure.

Imagine if the literal power company failed, and took days to tell people what was going on. You can see why people are reading the postmortem that exists, rather than the one that doesn't.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#188

Post by NicoJuicy » Sat, Nov 04, 2023, 7:05 PM UTC

ShadowRegent wrote: post #74

Interesting choice to spend the bulk of the article publicly shifting blame to a vendor by name and speculating on their root cause. Also an interesting choice to publicly call out that you're a whale in the facility and include an electrical diagram clearly marked Confidential by your vendor in the postmortem. Honestly, this is rather unprofessional. I understand and support explaining what triggered the event and g…

As far as I'm aware, this is the initial post-mortem to describe the events that took place.

And yes, that also means the initial event description in what they know so far.

Highly likely there will be another one https://twitter.com/eastdakota/status/1720688383607861442?t=...

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#189

Post by daxfohl » Sat, Nov 04, 2023, 7:13 PM UTC

disgruntledphd2 wrote: post #47

throwaway6920 wrote: post #33

Earlier quoted context omitted.

> In my experience, no engineers really decided to make services depend on just one data center. Partially true in this case; I can't speak to modern CF (or won't, moreso) but a large amount of internal services were built around SQL db's, and weren't built with any sense of eventual consistency. Usage of read replicas was basically unheard of. Knowing that, and that this was normal, it's a cultural issue rather than…

Dunno about that, I've read similar internal postmortems at the FAANG I worked at.

Everywhere I've worked requires a DR drill per service, but I've never seen anything where the whole company shuts down a DC at once across all services.

But probably we should. It's an immensely larger coordination problem, but frankly, it's probably the more common failure mode.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#190

Post by gpm » Sat, Nov 04, 2023, 7:18 PM UTC

AzzyHN wrote: post #185

To preface: I am not qualified to talk about this in the slightest. > Unfortunately, we discovered that a subset of services that were supposed to be on the high availability cluster had dependencies on services exclusively running in PDX-04 What do you mean you discovered? How could you not know? Surely when you were setting this high availability cluster up years ago and migrating services over, you double checked…

> And surely, since you had been "implementing" this for four years now, you've TESTED what would happen if one of the three DCs went completely offline, right???

They discussed this. They had been running tests were they disabled the high availability cluster in any of (and of two of) the three DCs. That test didn't involve disabling the rest of the (non-HA) services from PDX-04 DC (oops).