hot take: HN is way too biased and sympathetic towards the provider whenever an outage like this happens.
Post Mortem on Cloudflare Control Plane and Analytics Outage
41–50 of 241 posts
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#42Somewhat amazed at the structure of this article: after first discussing the third-party for 75% of blog post, the first-party recovery efforts were detailed in considerably lesser paragraphs. It’s promising to see a path forward mentioned but I can’t help but wonder why this was published instead of currently acknowledging their failure/circumstances and later on publishing a complete post-mortem after the dust full…
It’s called “shifting the blame”.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#43Cloudflare's control plane and analytics systems run primarily on servers in three data centers around Hillsboro, Oregon. The three data centers are independent of one another, each have multiple utility power feeds, and each have multiple redundant and independent network connections. The facilities were intentionally chosen to be at a distance apart that would minimize the chances that a natural disaster would caus…
They would then go into very detailed description how tectonic activity caused the outage.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#44Why the very first step was not to fail over Europe?
My question too, although possibly it seemed as a greater risk first to fail over. BTW, is there any unexpected GDPR implication of that? Assuming that fail over means restoring US backups in EU.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#45Earlier quoted context omitted.
? Tbh. As far as I can see, their data plane worked at the edge. Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush. Their control plane was bad though. But since most products worked, that's more redundancy than most products. The proposed solution is simple: - GA requires to be in the high availability cluster - test entire DC outages
> Tbh. As far as I can see, their data plane worked at the edge. Arguable, it's best to think of the edge as a buffering point in addition to processing. Aggregation has to happen somewhere, and that's where shit hit the fan.
? That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement.
Cloudflare's data lives in the edge and is constantly moving.
The only thing not living in the edge ( as was noticed), is stream, logpush and new image resize requests ( existing ones worked fine) from the data plane
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#46Well, this is definitely NOT a blameless post-mortem!
Obviously I'm joking because they are blaming an external company (Flexential) to which they are surely paying big money for the DC space.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#47Earlier quoted context omitted.
> why they decided to make services depend on just one data center In my experience, no engineers really decided to make services depend on just one data center. It happened because the dependency was overlooked. Or it happened because the dependency was thought to be a "soft dependency" with graceful degradation in case of unavailability but the graceful degradation path had a bug. Or it happened because the enginee…
> In my experience, no engineers really decided to make services depend on just one data center. Partially true in this case; I can't speak to modern CF (or won't, moreso) but a large amount of internal services were built around SQL db's, and weren't built with any sense of eventual consistency. Usage of read replicas was basically unheard of. Knowing that, and that this was normal, it's a cultural issue rather than…
Dunno about that, I've read similar internal postmortems at the FAANG I worked at.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#48> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discover…
> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet.
Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment!
> We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to innovate quickly. As such, products often take different paths toward their initial alpha. While, over time, our practice is to migrate the backend for these services to our best practices, we did not formally require that before products were declared generally available (GA). That was a mistake as it meant that the redundancy protections we had in place worked inconsistently depending on the product.
Complete and utter management failure. And customers apparently are sold what Cloudflare internally considers to be alpha quality software?
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#49> Our team was all-hands-on-deck and had worked all day on the emergency, so I made the call that most of us should get some rest and start the move back to PDX-04 in the morning. That decision delayed our full recovery, but I believe made it less likely that we’d compound this situation with additional mistakes.
I liked this - the human element is underemphasised often in these kinds of reports, and trying to fix a major outage while overly tired is only going to add avoidable mistakes.
I don’t know how it would work for an org of Cloudflare’s size, but I know we have plans for a significant outage for staff to work/sleep in shifts, to try to avoid that problem as well.
Issue there is that you need a way to hand over the current state of the outage to new staff as they wake up/come online.
Re: Post Mortem on Cloudflare Control Plane and Analytics Outage
#50> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discover…
And that this was unironically written in the same post mortem: “We are good at distributed systems.”
There’s a lack of awareness there.