Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

171–180 of 211 posts

Re: Cloudflare API Down

#171
post #37

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

> I have no idea why so many folks think large swaths of the Internet should be reliant on a single company. I don't think it's really that so many people think Cloudflare should be relied on. It's that Cloudflare generally has a good track record and their basic services are available for free. Actually, I don't know of any service similar to Cloudflare with such a generous free tier.

I think you're underestimating just how many devops and infra people dip into the mainstream tech news and adopt that as their new religion.

I have had multiple conversations with high level individuals about why we should be using CloudFlare so widely and what the fallback is if there is an outtage. Usually it boils down to "because reasons".

Today it's #notmyproblem and life is great.

Re: Cloudflare API Down

#172
I managed to log in last night but there wasn't much use to doing so. Once I was in I could barely run a domain search in the registration app. Errors were being thrown on every dashboard.

Re: Cloudflare API Down

#173
post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Had something similar happen at a telecom I worked at for years. We had a diesel generator and a couple of (bathroom sized) rooms full of (what looked like) car batteries. My understanding is that the two rooms were for redundancy. The batteries could power the DC for hours but were used only until the generator was ready.

The area our DC was located in was impressively reliable power-wise and -- in fact -- the backup systems had managed through the multi-state power outage in the early 2000s without a hitch (short of nearly running out of fuel due to our fuel supplier being ... just a little overwhelmed).

A few years later a two minute power outage caused the DC to go dark for a full day. Upon the power failing, the batteries kicked in and a few minutes after that the generator fired up and the DC went into holy terror.

About a minute after the generator kicked in, power to the DC blinked and ended. The emergency lights kicked in, the evacuate alarm sounded[0] and panic ensued.

My very pedestrian understanding of the problem was that a few things failed -- when the generator kicked in, something didn't switch power correctly, then something else didn't trip in response to that, a set of 4 batteries caught fire (and destroyed several nearby). They were extinguished by our facilities manager with a nearby fire extinguisher. He, incidentally, was the one who pulled the alarm (which wouldn't, on its own, trigger the Halon system, I think). The remainder of the day was spent dealing with the aftermath.

We were a global multi-national telecom with a mess of procedures in place for this sort of thing. Everything was installed by electricians, to very exacting standards[1] but -- as with most things "backup" -- the way it was tested and the frequency of those tests was inadequate.

From that point forward (going on over a decade) they thoroughly tested the battery/generator backup once a quarter.

[0] We were warned to GTFO if that alarm goes off due to the flooding of chemicals that would follow a few minutes later. That didn't happen.

[1] I remember the DC manager taking over in Cleveland making his staff work weeks of overtime replacing zip ties with wax lace (and it was done NASA style). We're talking thousands and thousands of runs stretching two complete floors of a skyscraper.

Re: Cloudflare API Down

#175
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

As of a few years ago, tiered cache and Argo are different things. I only wrote one of them, but in the marketing they are always referred to together. I think other than ENT users who can customize their tiered cache topology using the API it should be totally fine for the tiered cache topology to be days or weeks out of date. A lot of other stuff will have trouble, as you've correctly noted.

Re: Cloudflare API Down

#176

Earlier quoted context omitted.

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

These experiences of power outages is weird to me. What I consider "typical" data center design should make it really hard to lose power. "Typical" design would be: Each cabinet fed by 2 ATS (transfer switch). Each ATS fed by two UPS (battery bank). Each UPS fed by utility with generator backup. The two ATS can share one UPS/generator, so each cabinet would be fed by 3 UPS+generator. A generator failing to start shou…

You summed up quite well how these things happen.

All of these parts make for an increasingly complex system with a large number of failure points.

Our DC was a very living entity -- servers were being changed out/rack configuration altered very regularly. Large operations were carefully planned. You wouldn't overlook the power requirements of a few racks being added -- there'd -- literally[0] -- be no place to plug them in without an electrician being brought in. However, in a 3-month period every once in a while, two racks would have old devices replaced either due to failure or refresh, one at a time.

Since they weren't plugged directly into rack batteries (we had two battery rooms providing DC-wide battery backup), the overload wouldn't trip. Since we were still below the capacity of the circuit, the breaker(s) wouldn't trip. And maybe we're still under capacity for our backup system, but a few of the batteries are under-performing.

I think the lesson we learned when this happened was: you need to "actually test" the thing. My understanding is that our tests were of the individual components in isolation. We'd load test the batteries and the generator and then the relays between. At the end of the day, though, if you don't cut the power and see what happens you don't truly know. And my understanding is that having that final step in place resulted in a large number of additional tests being devised "of the individual components" that ensured they never had an outage like that, again.

[0] Guessing it's common practice to make "finding a f!cking power outlet" nearly impossible in DC. Every rack had exactly the number of leads it needed for the hardware plugged into a completely full receptacle. They rolled around a cart with a monitor, printer, label printer, keyboard, mouse and a huge UPS on it so staff could do daily maintenance work.

Re: Cloudflare API Down

#177
post #139

Earlier quoted context omitted.

I feel the same way. What about Akamai, Fastly, or Okta? Maybe Cloudflare gets more attention because their low end plans are accessible to anyone.

Its not just low end plans. Their pricing is basically the only one that feels fair. They don't charge you for bandwidth, unlike others that try to make on it as much as possible, while at the same time having other services also priced significantly higher.

They charge for bandwidth if you use enough of it on the enterprise tier.

Re: Cloudflare API Down

#178

Earlier quoted context omitted.

> " I have no idea why so many folks think large swaths of the Internet should be reliant on a single company. " Who thinks that? Can you link to anyone who has said that? Downvoted for "I am superior to " comment.

> Downvoted for "I am superior to " comment. I didn't interpret their comment this way. To me, it read "this thing gives me bad vibes and I don't understand why so many people like it."

"I don't understand why people like it" is very different from "so many folks think large swaths of the Internet should be reliant on a single company". Take Chrome browser; it's fine to use FireFox because you think it's better, it's fine to use FireFox even though you think it isn't as good but you'll take the mild inconvenience on the principle that the internet shouldn't be dependent on a single company. It's also fine to use Chrome because you think it's better, but nobody - absolutely nobody, anywhere, ever[1] - who chooses Chrome does so because they think the internet should be reliant on a single company for a web browser and they want to support making that happen.

"I don't use Chrome, I don't know why everyone thinks the internet should depend on Google" is a strawman because nobody does think that; many people use Chrome despite thinking the exact opposite of that, even. Same with CloudFlare, it's free, it's convenient, it's very good at what it does (current outage excepted), it's widely known, easy to work with, has good support. Nobody who chooses it does so because they want to hand internet control to a single company. And "I don't know why people use (popular, well known, well made solution)" is a very common internet comment which communicates a certain message.

[1] people who work for Google are paid to think that, so their decision doesn't count.

Re: Cloudflare API Down

#179
post #53
post #27

Did someone re-commit this bug from the other day? https://blog.cloudflare.com/cloudflare-incident-on-october-3... E: nah, looks like a power outage in their brain DC. See other comments for details :)

Every time CF goes down, we get rewarded with an interesting article. At this point I’ve been pavloved into enjoying outages!

Hah, right! I do love an in-depth postmortem
Post reply on HN