Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

61–70 of 211 posts

Re: Cloudflare API Down

#61
post #4

We use Cloudflare WARP (which is down) to access GitHub. Time to slack off.

If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...

I know git claims to be decentralized, but has anybody ever actually managed to use it in a decentralized manner? Not even the git or Linux projects themselves are without a centralized sync point.

Re: Cloudflare API Down

#62
post #41

Earlier quoted context omitted.

No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

You could do this with anything. X hasn't happened to me, so I bet it doesn't happen to other people, so people who take measures against X are misinformed/cargo-culting (unlike me who is conveniently the smart one in my narrative).

Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop. I default to Cloudflare because it's the only one that actually covers my ass.

Cloudflare is so successful because the internet was built naively as if abusers would never exist. Just consider how IP address spoofing is still possible today and you'll begin to realize how broken the internet has always been long before you even get into dirt cheap residential smart toaster botnets.

Re: Cloudflare API Down

#64

Earlier quoted context omitted.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

You could do this with anything. X hasn't happened to me, so I bet it doesn't happen to other people, so people who take measures against X are misinformed/cargo-culting (unlike me who is conveniently the smart one in my narrative). Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop.…

Please do take note that I'm not saying anything like what you claim I said. I'm asking if it's something people commonly get hit by, as I myself haven't had severe issues with it.

I'm not saying others are misinformed or cargo-culting anything, just that I'm seeing lots of people who probably never get hit by DDoS in the first place (couple of visitors per day) adding CloudFlare by default as that's what they see everyone else is doing.

Of course if you do frequently get hit by DDoS attacks, there is nothing wrong with trying to protect yourself against it...

Re: Cloudflare API Down

#65
post #50
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

The more concerning issue here is that their control plane is based out of a single datacenter.

A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.

Re: Cloudflare API Down

#66
post #41

Earlier quoted context omitted.

No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

Depends what you're counting as "little", but maybe your experience of 10 tiny sites has blinded you to the fact that sites for activist organisations, whistleblowing, investigative journalism, non-profits and so on, are very regularly targeted.

Re: Cloudflare API Down

#67

Earlier quoted context omitted.

If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...

I know git claims to be decentralized, but has anybody ever actually managed to use it in a decentralized manner? Not even the git or Linux projects themselves are without a centralized sync point.

Grandparent can still work off their local git repository, create local branches and commits, unlike the traditional VCS model which required branching and commits to be immediately synced to a centralized node.

The centralization issue that you raise is a different one; most projects intend to take contributions from folks and merge them into a single product.

Re: Cloudflare API Down

#68
post #50
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage."

Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Re: Cloudflare API Down

#69
post #51
post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

Yikes. If still true, this feels like a significant single point of failure in their architecture.

IIRC there is supposed to be a failover to AMS.

Re: Cloudflare API Down

#70
post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.
Post reply on HN