Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

Post reply on HN
Print view View original

121–130 of 211 posts

Re: Cloudflare API Down

#121

Post by jrockway »

nik736 wrote: post #70
gemstones wrote: post #68

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop.

Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.

Re: Cloudflare API Down

#122

Post by williamstein »

jrockway wrote: post #121
nik736 wrote: post #70

Earlier quoted context omitted.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.

One challenge is that the power usage of a server is order(s) of magnitude greater than that of a laptop. This means the cost to do what you describe is significant, hence that has to be taken into account when trying to build a cluster that is competitive...

Re: Cloudflare API Down

#123

Post by NicoJuicy »

is_true wrote: post #119
ryandvm wrote: post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

I feel they have one of the most fair offerings, not trying to squeeze every last penny from you like other cloud providers. Just my opinion of them, probably wrong.

I don't think you're wrong.

Reference: CEO takes less than average salary... 600 k./yr. After > 14 years.

Note: as far as I know ( which is one of the many reasons I do "believe" in cloudflare), he is already wealthy and didn't start cloudflare for the money.

The facts seem to be supportive of that statement ( cfr. Compensation).

The previous time I was a fan of a company was AMD in 2014 ( somewhere around that time) and because of Lisa Su + product line up.

Re: Cloudflare API Down

#124

Post by jrockway »

williamstein wrote: post #122
jrockway wrote: post #121

Earlier quoted context omitted.

I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.

One challenge is that the power usage of a server is order(s) of magnitude greater than that of a laptop. This means the cost to do what you describe is significant, hence that has to be taken into account when trying to build a cluster that is competitive...

Yeah, I agree with that. I think that power savings are a big priority for datacenters these days, so perhaps as more efficient chips go into production, the feasibility of "self-contained" servers increases. I could serve a lot of websites from my phone, and I've never had to fire up a diesel generator to have 100% uptime on it. (But, the network infrastructure uses more power than my phone itself. ONT + router is > 20W! The efficiency has to be everywhere for this to work.)

Re: Cloudflare API Down

#125

Post by acdha »

jabart wrote: post #100
greyface- wrote: post #32

Earlier quoted context omitted.

https://puck.nether.net/pipermail/outages/2023-November/0149... > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet).

Our local test those like once a month. Looks like our account manager will get a quick check in after this settles to make sure we won't have this issue.

Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor.

The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generator because the main fuel tank pump flooded (AWS was completely unaffected). You just don’t tempt the DR fates like that…

Re: Cloudflare API Down

#126

Post by jabart »

acdha wrote: post #125
jabart wrote: post #100

Earlier quoted context omitted.

Our local test those like once a month. Looks like our account manager will get a quick check in after this settles to make sure we won't have this issue.

Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor. The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generat…

Same. Local company had a NOC and power went out in the area. They had moved into the build a long time ago and had two backup generators at the back end of the parking lot already installed. Power goes out, generator didn't start. 2nd generator started but it was now undersized and immediately tripped the breaker. First generator tried to start and it failed again.

Another local company had a backup generator in the basement. They were doing maintenance work on the power and messed up so the generator kicked in. Since it was in the basement, they had a belt to drive an exhaust fan. Generator kicked on, fan belt snapped, exhaust filled the entire basement, rolled into the service elevator, starts coming out the 3rd floor (top of the service elevator) and the building was then evacuated.

Last generator story, small IT shop with a small colo in the 2000s moved into a new building and got a natural gas generator installed with it running a self diagnostic once a month. Power goes out in the first month, lights go out, generator kicks on, lights come back, lights go back off. Turns out that the electrician didn't have a 60amp breaker that day and put in a 20 amp breaker for a 60+ amp generator. They meant to come back out and fix it and never did.

Re: Cloudflare API Down

#127

Post by acdha »

jabart wrote: post #126
acdha wrote: post #125

Earlier quoted context omitted.

Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor. The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generat…

Same. Local company had a NOC and power went out in the area. They had moved into the build a long time ago and had two backup generators at the back end of the parking lot already installed. Power goes out, generator didn't start. 2nd generator started but it was now undersized and immediately tripped the breaker. First generator tried to start and it failed again. Another local company had a backup generator in the…

> Generator kicked on, fan belt snapped, exhaust filled the entire basement, rolled into the service elevator, starts coming out the 3rd floor (top of the service elevator) and the building was then evacuated.

I feel like a ton of these stories come down to infrequent use. I remember one where they’d tested the generator but only for like a couple of minutes each month and so the fuel in the tank was really old by the time there was a real outage and the generator clogged expensively about an hour in, right as people were relaxing and thinking it’d be smooth sailing until the main power line was restored.

Re: Cloudflare API Down

#128

Post by slig »

dharmab wrote: post #96
supriyo-biswas wrote: post #88

Earlier quoted context omitted.

This can just be multiple datacenters located close together (~100 km) similar to AWS AZs.

Fun fact, on certain (major) cloud providers, in certain regions, AZs are sometimes different floors of the same building :)

The latency is exceptional.

Re: Cloudflare API Down

#129

Post by AtNightWeCode »

jrockway wrote: post #121
nik736 wrote: post #70

Earlier quoted context omitted.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.

They most likely lie about the power outage. Azure does this all the time. I am so fking tired of these services. Even minor data centers in Europe have backup plans for power.

Re: Cloudflare API Down

#130

Post by barkingcat »

gemstones wrote: post #68
flutas wrote: post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Datacentre administrators don't know how to run utilities.

Imagine replacing the word "power" with "sewage" and try to see if you would entrust the functionality of your toilet to your local friendly sysadmin.

No. You'd never ask a system administrator to administer your plumbing. Neither should you ask your system administrator to maintain a diesel power generator. Diesel generators have more in common with automobile internal combustion engines and in the high power segment, airplane jet turbines. In fact, many turbine cores are used as both airplane jet engine and as terrestrial power generation unit.

You're basically asking the wrong people to maintain the infrastructure.

Post reply on HN