Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

151–160 of 211 posts

Re: Cloudflare API Down

#151
post #70

Earlier quoted context omitted.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

I wonder why we don't put battery backups in each server/switch/etc. Basically, just be a laptop in each 1U rack space instead of a desktop. Sure, you can't have much runtime, but if you got like 15 minutes for each device and it always worked, you could smooth over a lot of generator problems when something chews through the building's main grid connection.

It’s pretty common to have a rack of batteries that might serve an isle. The idea of these is that you’d have enough juice for the generator to kick in. You couldn’t run these for longer periods, and even if you could, you’d still have the AC unpowered, which would quickly lead to machines overheating and crashing. Plus the building access controls need powering too. As does lighting, and a whole host of other critical systems. But the AC alone is a far more significant problem than powering the racks. (I’ve worked in places when the AC has failed, it’s not fun. You’d be amazed how much heat those systems can kick out).

Re: Cloudflare API Down

#152

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Test your backups! Obviously easier said than done of course.

Experience in 'small' high availability safety-critical systems says:

1- 'failover often, failover safely'. Things that run once a month or 'just in case' are the most likely to fail.

2- people (customers) often aren't ready to pay for the cost of designing and operating systems with the availability levels they want.

Re: Cloudflare API Down

#153
post #40
post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

In principle I agree with this, but do feel this is said more readily about Cloudflare than other companies it could said about - such as Amazon (via AWS), Google and Microsoft. Perhaps my own mental model is wrong, but I see them as a credible challenger to those very oligopolistic companies, and wish there were more Cloudflares.

The difference is that nobody complains and most people agree when you talk smack about Amazon, Google and Microsoft. The general consensus is that they're big, dumb and knowingly evil, and most of the time their actions can be explained by that.

When we talk smack about Cloudflare, such as about their hosting of phishing, their underhanded DoH stuff, their complete lack of abuse handling, et cetera, lots of people come to their defense and make excuses for them.

You can like a company's product and still think the company is big and desires to be evil, but there's an emotional component for some that makes "us versus them" knee-jerk reactions more compelling than, "hmmm... is this correct?" evaluations.

I don't think any of these Cloudflare apologists would try to argue on facts that Cloudflare isn't trying to be a monopoly, isn't trying to recentralize the Internet, isn't marginalizing the rest of the non-western world, isn't trying to establish dependencies that people and companies can't easily escape, but if they did, that'd make for some interesting discussion.

Re: Cloudflare API Down

#154

Earlier quoted context omitted.

Which can be said from any cloud provider/hoster.

How so? If I'm hosting a server somewhere and clients directly connect to my server to establish a TLS connection, failing any vulnerabilities in the implementation, there's no MITM happening and the provider can't see the plaintext traffic. (Of course, since the server needs the certificate, the provider could in theory extract that certificate and establish a MITM proxy, but this isn't by design.)

Any VPS or virtual server cloud provider can potentially see the plaintext traffic - it's in plain text of the memory of their hardware and they could be looking at it. They technically could be scraping your SSL keys from memory, or scraping your SSL private key from disk (if unencrypted storage) and then decrypting a mirror of the network traffic elsewhere. That wouldn't be MITM but you are only protected from it if you are hosting your own physical server somewhere.

"End to end security" mentioned above is limited security when "your" endpoint is owned by and controlled by someone else.

Re: Cloudflare API Down

#155
post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

> "I have no idea why so many folks think large swaths of the Internet should be reliant on a single company."

Who thinks that? Can you link to anyone who has said that?

Downvoted for "I am superior to " comment.

Re: Cloudflare API Down

#156
post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

> " I have no idea why so many folks think large swaths of the Internet should be reliant on a single company. " Who thinks that? Can you link to anyone who has said that? Downvoted for "I am superior to " comment.

> Downvoted for "I am superior to " comment.

I didn't interpret their comment this way. To me, it read "this thing gives me bad vibes and I don't understand why so many people like it."

Re: Cloudflare API Down

#157

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Datacentre administrators don't know how to run utilities. Imagine replacing the word "power" with "sewage" and try to see if you would entrust the functionality of your toilet to your local friendly sysadmin. No. You'd never ask a system administrator to administer your plumbing. Neither should you ask your system administrator to maintain a diesel power generator. Diesel generators have more in common with automobi…

When I worked in a DC the HVAC guys did the cooling. The electricians did the power and genset. We also had a local GE guy who did the engine part of the genset. These aren't sysadmin running generators. They are specialists hired for the job.

Re: Cloudflare API Down

#158
post #71

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Lack of preventive maintenance if I were to guess. Also, these generators would need a supply of diesel fuel, and typically have a storage tank on site. If the diesel isn't used and replaced, it can gum up the generator.

I've gotten 60 year old tractors to run on 60 year old diesel. Gumming up is much more common in gas applications. I guess modern diesel might not be so robust, I know almost nothing about modern engines.

Re: Cloudflare API Down

#159
post #150
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

What is PDX?

Cloudflare's Portland datacenter. Most clouds/CDNs name their DC's after the airport code of the city they are in or are close to. Internally they would be listed as PDX, ORD, LAX, etc.

Re: Cloudflare API Down

#160
post #150
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

What is PDX?

Portland Data Center I believe
Post reply on HN