Live data from Hacker News

Cloudflare API Down

cloudflarestatus.com

91–100 of 211 posts

Re: Cloudflare API Down

#91

Earlier quoted context omitted.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

You could do this with anything. X hasn't happened to me, so I bet it doesn't happen to other people, so people who take measures against X are misinformed/cargo-culting (unlike me who is conveniently the smart one in my narrative). Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop.…

> Just consider how IP address spoofing is still possible today and you'll begin to realize how broken the internet has always been long before you even get into dirt cheap residential smart toaster botnets.

I know I'm going to regret asking but, ok, I'll bite... why does IP address spoofing prove the Internet is broken? Especially considering that a) the point of internet routing is to route packets whenever possible especially around damage. and b) by volume, the internet is TCP and you can't complete a handshake with a spoofed ip.

Re: Cloudflare API Down

#92

Not just the API - I'm now suddenly having DNS troubles with one of my domains. I think maybe only domains/sites that use their "configuration rules", or some specific feature like that.

Happened to me yesterday

Re: Cloudflare API Down

#93
post #41

Earlier quoted context omitted.

No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

good luck with all those strawman replies...

It only takes one trivial event to make you a cloudflare zealot, most ISPs (web hosts) don't provide attack mitigation nor prevention. If you didn't know that you could ban IPs programtically you'd have been screwed* and would have loved how quickly cloudflare would protect your site.

It's not a difficulty thing, it's an ignorance thing. cloudflare isn't better,or magic, it's easy and popular. Just like social media, and aws.

For some people, i.e. most 'web developers' you will get hacked if you don't use it... because the alternative would be literally nothing. Not everyone wants to do things correctly, they only want it to work, working correctly is required.

Re: Cloudflare API Down

#94

Earlier quoted context omitted.

You could do this with anything. X hasn't happened to me, so I bet it doesn't happen to other people, so people who take measures against X are misinformed/cargo-culting (unlike me who is conveniently the smart one in my narrative). Most services I've built that achieved any sort of traction have dealt with some sort of DoS including large fees when I've used CDNs like Cloudfront that are susceptible to a wget loop.…

> Just consider how IP address spoofing is still possible today and you'll begin to realize how broken the internet has always been long before you even get into dirt cheap residential smart toaster botnets. I know I'm going to regret asking but, ok, I'll bite... why does IP address spoofing prove the Internet is broken? Especially considering that a) the point of internet routing is to route packets whenever possibl…

With spoofed src addr:

* you can do all sorts of udp amplification attacks (e.g. dns - send a zone transfer request in a single packet with a spoofed IP, and the IP you spoofed to gets a lot of traffic in response.)

* you can do tcp syn or ack floods with a spoofed IP, these eat resources on the target machine. syn floods cause the os to allocate a new connection and timers waiting for the third ack.

* you can send lots of bad packets from a spoofed ip that causes automated systems to lock out those IPs as a response to attack traffic. If those lockouts block IPs that should be allowed (a type of denial of service)

And plenty more.

Re: Cloudflare API Down

#95
post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Running a generator yard is just hard. You are acting as your own power utility with equipment that only runs during tests or outages. Running successfully at commissioning or during tests increases likelihood of service when needed, but is not a guarantee.

https://aws.amazon.com/message/67457/ (AWS: Summary of the AWS Service Event in the US East Region, July 2, 2012)

> On Friday night, as the storm progressed, several US East-1 datacenters in Availability Zones which would remain unaffected by events that evening saw utility power fluctuations. Backup systems in those datacenters responded as designed, resulting in no loss of power or customer impact. At 7:24pm PDT, a large voltage spike was experienced by the electrical switching equipment in two of the US East-1 datacenters supporting a single Availability Zone. All utility electrical switches in both datacenters initiated transfer to generator power. In one of the datacenters, the transfer completed without incident. In the other, the generators started successfully, but each generator independently failed to provide stable voltage as they were brought into service. As a result, the generators did not pick up the load and servers operated without interruption during this period on the Uninterruptable Power Supply (“UPS”) units. Shortly thereafter, utility power was restored and our datacenter personnel transferred the datacenter back to utility power. The utility power in the Region failed a second time at 7:57pm PDT. Again, all rooms of this one facility failed to successfully transfer to generator power while all of our other datacenters in the Region continued to operate without customer impact.

> The generators and electrical switching equipment in the datacenter that experienced the failure were all the same brand and all installed in late 2010 and early 2011. Prior to installation in this facility, the generators were rigorously tested by the manufacturer. At datacenter commissioning time, they again passed all load tests (approximately 8 hours of testing) without issue. On May 12th of this year, we conducted a full load test where the entire datacenter switched to and ran successfully on these same generators, and all systems operated correctly. The generators and electrical equipment in this datacenter are less than two years old, maintained by manufacturer representatives to manufacturer standards, and tested weekly. In addition, these generators operated flawlessly, once brought online Friday night, for just over 30 hours until utility power was restored to this datacenter. The equipment will be repaired, recertified by the manufacturer, and retested at full load onsite or it will be replaced entirely. In the interim, because the generators ran successfully for 30 hours after being manually brought online, we are confident they will perform properly if the load is transferred to them. Therefore, prior to completing the engineering work mentioned above, we will lengthen the amount of time the electrical switching equipment gives the generators to reach stable power before the switch board assesses whether the generators are ready to accept the full power load. Additionally, we will expand the power quality tolerances allowed when evaluating whether to switch the load to generator power. We will expand the size of the onsite 24x7 engineering staff to ensure that if there is a repeat event, the switch to generator will be completed manually (if necessary) before UPSs discharge and there is any customer impact.

Re: Cloudflare API Down

#96
post #77

Earlier quoted context omitted.

Kubernetes' default datastore, etcd, is not tolerant of latencies between multiple regions. Generally, vanilla k8s clusters have a single-region control plane.

This can just be multiple datacenters located close together (~100 km) similar to AWS AZs.

Fun fact, on certain (major) cloud providers, in certain regions, AZs are sometimes different floors of the same building :)

Re: Cloudflare API Down

#97
post #41

Earlier quoted context omitted.

No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.

How many little sites do you run that get hit by DDoS? I personally run about 10 tiny websites myself, some of them have around ~1-2K daily active users, but neither of them have suffered from any DDoS frequently nor do they use CloudFlare at all. One has been hit once by a DDoS that kept trying for ~2 days to bring the site down, but a simple "ban IPs based on hitting rate limits" did the trick to avoid going down,…

> How many little sites do you run that get hit by DDoS?

Do you live in a country that has enemies? DDoS attacks are one of the primary weapons of cyberwarfare.

You may not have suffered an attack yet, but thinking the world is a peaceful place in which small players have no worries is naïve.

Re: Cloudflare API Down

#98
post #50

Earlier quoted context omitted.

Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in p…

I think every datacenter I've ever worked with, across ~4 jobs, has had an incident report like "generator failed right as we had an outage." Am I unlucky, or is there something I miss about datacenter administration that makes it really hard to maintain a generator? I guess you don't hear about times the generator worked, but it feels like a high rate of failure to me.

Test your backups! Obviously easier said than done of course.

Re: Cloudflare API Down

#99
post #73
post #70

Earlier quoted context omitted.

Even the high profile datacenters I had to deal with in Frankfurt had the same issues. There were regular maintenance tests where they made sure the generators were working properly... I can imagine this is more of a pray and sweat task than anything that's in your hands. I have no clue why this is the status quo though.

The phone utility were I live has deisel generators that kick on whenever the power goes out in order to keep the copper phone lines operational. These generators always work, or at least one of the four they have in each office does.

The datacenter I was in for awhile had the big gens, and with similar "phone utility" setups - they would cut to the backup gens once a month and run for longer than the UPS could hold the facility (if they detected an issue, they'd switch back to utility power).

They also had redundant gensets (2x the whole facility, 4x 'important stuff' - you could get a bit of a discount by being willing to be shut off in a huge emergency where gens were dying/running out of fuel).

Re: Cloudflare API Down

#100
post #25

Earlier quoted context omitted.

Yikes, I'm curious how that happened. Our Flexential has 5 generators and a basement full of battery backups along with isolated zones.

https://puck.nether.net/pipermail/outages/2023-November/0149... > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet).

Our local test those like once a month. Looks like our account manager will get a quick check in after this settles to make sure we won't have this issue.
Post reply on HN