We use Cloudflare WARP (which is down) to access GitHub. Time to slack off.
If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
Cloudflare API Down
61–70 of 211 posts
Re: Cloudflare API Down
#62Earlier quoted context omitted.
No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.
How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…
Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop. I default to Cloudflare because it's the only one that actually covers my ass.
Cloudflare is so successful because the internet was built naively as if abusers would never exist. Just consider how IP address spoofing is still possible today and you'll begin to realize how broken the internet has always been long before you even get into dirt cheap residential smart toaster botnets.
Re: Cloudflare API Down
#63Re: Cloudflare API Down
#64Earlier quoted context omitted.
How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…
You could do this with anything. X hasn't happened to me, so I bet it doesn't happen to other people, so people who take measures against X are misinformed/cargo-culting (unlike me who is conveniently the smart one in my narrative). Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop.…
I'm not saying others are misinformed or cargo-culting anything, just that I'm seeing lots of people who probably never get hit by DDoS in the first place (couple of visitors per day) adding CloudFlare by default as that's what they see everyone else is doing.
Of course if you do frequently get hit by DDoS attacks, there is nothing wrong with trying to protect yourself against it...
Re: Cloudflare API Down
#65When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…
Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…
A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.
Re: Cloudflare API Down
#66Earlier quoted context omitted.
No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.
How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…
Re: Cloudflare API Down
#67Earlier quoted context omitted.
If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
I know git claims to be decentralized, but has anybody ever actually managed to use it in a decentralized manner? Not even the git or Linux projects themselves are without a centralized sync point.
The centralization issue that you raise is a different one; most projects intend to take contributions from folks and merge them into a single product.
Re: Cloudflare API Down
#68When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…
Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…
Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.
Re: Cloudflare API Down
#69When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…
Yikes. If still true, this feels like a significant single point of failure in their architecture.
Re: Cloudflare API Down
#70Earlier quoted context omitted.
Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…
I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.