Cloudflare API Down

Post by thedaly » Thu, Nov 02, 2023, 2:40 PM UTC

gemstones wrote: post #68

flutas wrote: post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Post by thedaly » Thu, Nov 02, 2023, 2:43 PM UTC

nik736 wrote: post #70

gemstones wrote: post #68

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

Post by handsclean » Thu, Nov 02, 2023, 2:45 PM UTC

supriyo-biswas wrote: post #67

handsclean wrote: post #61

Earlier quoted context omitted.

I know git claims to be decentralized, but has anybody ever actually managed to use it in a decentralized manner? Not even the git or Linux projects themselves are without a centralized sync point.

Grandparent can still work off their local git repository, create local branches and commits, unlike the traditional VCS model which required branching and commits to be immediately synced to a centralized node. The centralization issue that you raise is a different one; most projects intend to take contributions from folks and merge them into a single product.

Post by dharmab » Thu, Nov 02, 2023, 2:50 PM UTC

supriyo-biswas wrote: post #65

flutas wrote: post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.

Post by acejam » Thu, Nov 02, 2023, 2:57 PM UTC

ebbp wrote: post #40

ryandvm wrote: post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

In principle I agree with this, but do feel this is said more readily about Cloudflare than other companies it could said about - such as Amazon (via AWS), Google and Microsoft. Perhaps my own mental model is wrong, but I see them as a credible challenger to those very oligopolistic companies, and wish there were more Cloudflares.