Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem on Cloudflare Control Plane and Analytics Outage

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 241 posts

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#111

Post by belter »

davewritescode wrote: post #95
creshal wrote: post #48

Earlier quoted context omitted.

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment! > We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to inn…

I’m going to leave out some details but there was a period of time where you could bypass cloudflare’s IP whitelisting by using Apple’s iCloud relay service. This was fixed but to my knowledge never disclosed.

There was a time when they were dumping encryption keys into search engine caches for weeks, and had the audacity to claim here, the issue was "mostly" solved. Until they were called out on it by Google Project Zero team...

"Cloudflare Reverse Proxies Are Dumping Uninitialized Memory" - https://news.ycombinator.com/item?id=13718752

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#112

Post by byteknight »

davewritescode wrote: post #95
creshal wrote: post #48

Earlier quoted context omitted.

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment! > We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to inn…

I’m going to leave out some details but there was a period of time where you could bypass cloudflare’s IP whitelisting by using Apple’s iCloud relay service. This was fixed but to my knowledge never disclosed.

There still exist many bypasses that work in a lot of cases. There's even services for it now. Wouldn't be surprised if that or similar was a technique employed.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#113

Post by usr1106 »

Tocra wrote: post #40

I am really upset about this situation on behalf of CF however why don't they think about generating their own electricity with renewable energy sources?

And why not run their own DC?

It doesn't change anything fundamentally. A complex product is only as good as the weakest link. I have worked with various employers, some world leaders at the time. All of them had seriously weak links.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#114

Post by NicoJuicy »

martinald wrote: post #94
__turbobrew__ wrote: post #29

> However, we had never tested fully taking the entire PDX-04 facility offline. That is a painful lesson, but unless you are physically powering off the dc or at least disconnecting the network from the outside world you are not testing a real disaster. You can point fingers at the facility operators, but at the end of the day you have to be able to recover from a dc going completely offline and maybe never coming ba…

This is a fair point. Imagine there had been a serious fire like OVH suffered or flooding that destroyed the data center. Would Cloudflare have been able to recover?

That's not what happened here. Their edge worked fine.

Business was mostly running as usual.

The OVH outage was immediate downtime.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#115

Post by whoknowsidont »

>While there were periods where customers were unable to make changes to those services, traffic through our network was not impacted.

They're just going to straight up lie like that? We definitely weren't able to get "traffic through [their] network" through the outage at many different random points.

So if the CF team is under the impression traffic was not impacted, dig deeper.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#116

Post by belter »

steve1977 wrote: post #51
davedx wrote: post #50

Earlier quoted context omitted.

And that this was unironically written in the same post mortem: “We are good at distributed systems.” There’s a lack of awareness there.

Well, they did distribute their systems. Some were in the running DC, some were not ;)

They distributed the faults across all their customers....

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#117

Post by dopylitty »

solatic wrote: post #12

Not criticism, just remarks: > While, over time, our practice is to migrate the backend for these services to our best practices, we did not formally require that before products were declared generally available (GA). I really like the model where a single team in a company, with Product + Dev, can quickly ship, iterate on a new product, and prove market demand without going through layers and layers of internal bur…

> We need to use the distributed systems products that we make available to all our customers for all our services so they continue to function mostly as normal even if our core facilities are disrupted.

>> Super excited to see this. Cloudflare Workers is still too much of an "edge" platform and not a "main datacenter" platform, at least because D1 is still in beta and even if it wasn't, Postgres is far more feature-ful, and that pulls more software into a traditional single-datacenter model. So if Cloudflare can really succeed at this, then it'll be a much stronger statement in favor of building out software in an edge-only model.

On the other when a company dogfoods its own products you end up in a dependency hell like AWS apparently is in where a single Lambda cell hitting full capacity in us-east-1 breaks many services in all regions.

I'm sure there is a right way to manage end to end dependencies for 100% of your services past, present, and future but increasingly I'm of the opinion that it's not possible in our economic system to dedicate enough resources to maintain such a dependency mapping system since that takes away developer time from customer facing products that show up in the bottom line. You just limp along and hope that nothing happens that takes out your whole product.

Maybe companies whose core business is a money printing machine (ads) can dedicate people to it but companies whose core business is tech probably don't have the spare cash.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#119

Post by ianhawes »

cj wrote: post #105
someonehere wrote: post #103

I asked a sales rep once about services going out and how that would affect CF For Teams. They said it would be virtually impossible for CF to go down because of all their data centers around the world. Paraphrasing, “if there’s an outage, there’s definitely something going wrong with the internet.” And here we are. My trust in them has hit zero.

FWIW, I'm a Cloudflare Enterprise customer and we had zero downtime. Only thing that was temporarily unavailable was the cloudflare dashboard. I feel like a lot of people in this thread are commenting under the impression that all of Cloudflare was down for 24 hours when in reality I wouldn't be surprised if a lot of customers were unaffected and unaware of the incident. I wouldn't even have known of the outage had i…

2nd this. We had zero downtime on anything in production. The only reason we knew is because we are actively standing up a transition to R2 and ran into errors configuring buckets.

Re: Post Mortem on Cloudflare Control Plane and Analytics Outage

#120

Post by brookst »

creshal wrote: post #48
Dunedan wrote: post #21

> While most of our critical control plane systems had been migrated to the high availability cluster, some services, especially for some newer products, had not yet been added to the high availability cluster. > The other two data centers running in the area would take over responsibility for the high availability cluster and keep critical services online. Generally that worked as planned. Unfortunately, we discover…

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment! > We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to inn…

> Complete and utter management failure

Too strong. A failure certainly, but painting this as the worst possible management failure is kind of silly.

Post reply on HN