And yet their share price is up over 5% today
https://uk.finance.yahoo.com/quote/NET?.tsrc=applewf&guccoun...
Cloudflare API Down
181–190 of 211 posts
Re: Cloudflare API Down
#182Earlier quoted context omitted.
I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
Had something similar happen at a telecom I worked at for years. We had a diesel generator and a couple of (bathroom sized) rooms full of (what looked like) car batteries. My understanding is that the two rooms were for redundancy. The batteries could power the DC for hours but were used only until the generator was ready. The area our DC was located in was impressively reliable power-wise and -- in fact -- the backu…
I lost track of how many datacenter outages we caused testing the power backup/failover back at eBay in the mid-2000s.
There's no winning when it comes to power redundancy systems.
Re: Cloudflare API Down
#183It's now been 12 hours and they are still having issues. I'm taking an online video course in music production and they host their videos on cloudflare stream and none of them work. https://www.cloudflare.com/products/cloudflare-stream/ The single point of failure of half the internet continues to rear it's head.
> "These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge."
https://www.cloudflarestatus.com/incidents/hm7491k53ppg
I thought this only affected the Dashboard (and functionality inside the dashboard).
Not anything else.
Is that not the case, since you're saying cloud flare video streaming isn't working?
Re: Cloudflare API Down
#184Earlier quoted context omitted.
Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…
I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
It's not usually the battery backup or the generator that fails. It's usually the switching equipment that has to go from mains to battery to generator to battery to mains. And doing it without causing a voltage sag on the generator.
Re: Cloudflare API Down
#185Earlier quoted context omitted.
> Downvoted for "I am superior to " comment. I didn't interpret their comment this way. To me, it read "this thing gives me bad vibes and I don't understand why so many people like it."
"I don't understand why people like it" is very different from "so many folks think large swaths of the Internet should be reliant on a single company". Take Chrome browser; it's fine to use FireFox because you think it's better, it's fine to use FireFox even though you think it isn't as good but you'll take the mild inconvenience on the principle that the internet shouldn't be dependent on a single company. It's als…
You're right in the literal sense, but I don't think they meant that literally (maybe they did, who knows).
I personally haven't seen anyone praise the Internet being reliant on a single company. However, I have seen lots of praise for Cloudflare over the years and "naysayers" (such as people raising concerns about them MITMing half the Internet) being aggressively downvoted. In that context, I see it as many people tacitly endorsing Cloudflare and not caring about the control it holds, rather than people explicitly saying "Cloudflare _should_ control the majority of the Internet."
Re: Cloudflare API Down
#186Earlier quoted context omitted.
If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
Yes, it's also good a variety of different systems to store and address Markdown.
Which you could, you know, put in git.
Git is more than just GitHub though, that much is true.
Re: Cloudflare API Down
#187Earlier quoted context omitted.
Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
I would like to see Git be extended with a decentralized approach to bugtracking, code reviews, wiki's for documentation etc.
What do you need for wikis? Just put text files in a folder?
Bug tracking in git: https://github.com/MichaelMure/git-bug
Code review in git: https://github.com/google/git-appraise
Re: Cloudflare API Down
#188Earlier quoted context omitted.
I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
These experiences of power outages is weird to me. What I consider "typical" data center design should make it really hard to lose power. "Typical" design would be: Each cabinet fed by 2 ATS (transfer switch). Each ATS fed by two UPS (battery bank). Each UPS fed by utility with generator backup. The two ATS can share one UPS/generator, so each cabinet would be fed by 3 UPS+generator. A generator failing to start shou…
While what you're describing is definitely possible, but, datacenter architecture is becoming less and less bulletproof-reliable in service of efficiency (both cost as well as PUE).
Re: Cloudflare API Down
#189Earlier quoted context omitted.
I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
These experiences of power outages is weird to me. What I consider "typical" data center design should make it really hard to lose power. "Typical" design would be: Each cabinet fed by 2 ATS (transfer switch). Each ATS fed by two UPS (battery bank). Each UPS fed by utility with generator backup. The two ATS can share one UPS/generator, so each cabinet would be fed by 3 UPS+generator. A generator failing to start shou…
>> These experiences of power outages is weird to me. What I consider "typical" data center design should make it really hard to lose power.
At least 30% of datacenter outages that we had with a large company were due to some power related issues.
Just a simple small scale one: the technician accidentally plugged in the redundant circuits into the same source power link. When we lost a phase it took down the 2/3 of the capacity instead of 1/3. Hoops.
Re: Cloudflare API Down
#190Earlier quoted context omitted.
The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.
Kubernetes' default datastore, etcd, is not tolerant of latencies between multiple regions. Generally, vanilla k8s clusters have a single-region control plane.
We run a k8s control plane across datacenter in west, central, and east US and it works fine.