Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

Post reply on HN
Print view View original

81–90 of 211 posts

Re: Cloudflare API Down

#81

Post by sophacles »

capableweb wrote: post #45
remram wrote: post #41

Earlier quoted context omitted.

No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

I've never had any of my 3 houses burn down. I don't know why everyone says buy insurance, and having a fire extinguisher is a good idea.

It seems to me like a common misconception that you "have to have a fire extinguisher" (or other supression system), otherwise you'll have your house burn down.

Re: Cloudflare API Down

#83

Post by 015a »

buro9 wrote: post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

> Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover

IIRC, AMS was/is just a new DC to replace or augment LUX, which has existed for some time as, among other things, a failover secondary for PDX. But yeah, intention and reality drifted, as they always do; I had heard a variety of reasons why automated, or even break-glass, failover to a DC on the opposite side of the planet wasn't a tenable solution for many workloads.

They're big on chaos testing, and definitely (very regularly) test whole-DC network partitions. I'd love to be a fly on the wall in the meeting discussing why this event was so different.

Re: Cloudflare API Down

#85

Post by roamerz »

buro9 wrote: post #46

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…

Generators, AC units, UPS’s and auto cutover switches are all mechanical susceptible to failure. You can do your best with maintenance and pre planning but in the end there is a non zero chance that a condition that you didn’t anticipate or couldn’t afford to account for will happen and your critical infrastructure will go down.

Re: Cloudflare API Down

#86

Post by zozskuh »

tppiotrowski wrote: post #80
londons_explore wrote: post #44

Earlier quoted context omitted.

Perhaps someone was rushing to release something that was going to be announced on the earnings call...

That was yesterday. They announced pricing changes for Cloudflare workers. I think it won't affect me but heavy workloads will have to pay more. They used to charge a flat fee but now are moving to $/CPU time. Overall I still love Cloudflare and run all my backend stuff there. It just feels simpler and cleaner than AWS but it's slowly starting to get cluttered as more features are released.

"Cloudflare, Inc. Class A Common Stock is expected* to report earnings on 11/02/2023 after market close."

https://www.nasdaq.com/market-activity/stocks/net/earnings

Re: Cloudflare API Down

#87

Post by NicoJuicy »

tppiotrowski wrote: post #80
londons_explore wrote: post #44

Earlier quoted context omitted.

Perhaps someone was rushing to release something that was going to be announced on the earnings call...

That was yesterday. They announced pricing changes for Cloudflare workers. I think it won't affect me but heavy workloads will have to pay more. They used to charge a flat fee but now are moving to $/CPU time. Overall I still love Cloudflare and run all my backend stuff there. It just feels simpler and cleaner than AWS but it's slowly starting to get cluttered as more features are released.

?

They announced that on 28/09 fyi. It's not new and they didn't change a flat fee before.

They still have the 5 € / month fee.

https://blog.cloudflare.com/workers-pricing-scale-to-zero/

Re: Cloudflare API Down

#88

Post by supriyo-biswas »

dharmab wrote: post #77
supriyo-biswas wrote: post #65

Earlier quoted context omitted.

The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.

Kubernetes' default datastore, etcd, is not tolerant of latencies between multiple regions. Generally, vanilla k8s clusters have a single-region control plane.

This can just be multiple datacenters located close together (~100 km) similar to AWS AZs.

Re: Cloudflare API Down

#89

Post by alams »

Looks like Salesforce and Cloudflare sharing the same datacenter. Salesforce had the same issue today and they mentioned similar power failure as the root cause.

It is a known fact that salesforce is running its "Hyperforce" in AWS.

Does that mean Cloudflare also using AWS?

Does anyone faced any issue on AWS resources today?

Re: Cloudflare API Down

#90

Post by nacholibrev »

alams wrote: post #89

Looks like Salesforce and Cloudflare sharing the same datacenter. Salesforce had the same issue today and they mentioned similar power failure as the root cause. It is a known fact that salesforce is running its "Hyperforce" in AWS. Does that mean Cloudflare also using AWS? Does anyone faced any issue on AWS resources today?

Judging from the AWS status page it seems that all services are up and running without any issues.

https://health.aws.amazon.com/health/status

Post reply on HN