Live data from Hacker News

Cloudflare had a partial outage

cloudflare.com

291–300 of 448 posts

Re: Cloudflare had a partial outage

#291

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.

All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors.

The Cloudflare Dashboard is also no longer fully loading.

Re: Cloudflare had a partial outage

#292
post #92

At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.

Are there places to host an independent bare-metal box where the internet provider for that box is more reliable than cloudflare?

I have been running my business on Hetzner bare-metal servers for the last 7 years. During that time there were several brief network outages, on the order of minutes. I think one network outage was 30 minutes. Other than that, no problems.

Given the price and performance difference between bare-metal and everything else, I am puzzled as to why small businesses that do not need scalability do not go with bare metal. And given the speeds of todays hardware, if you are not doing something stupid and you have a B2B SaaS, it's really difficult to need "scalability" beyond several bare-metal servers.

To be clear, I do not consider my bare-metal boxes "reliable", I have a multi-server setup managed by ansible, with a distributed database, and I can take a single-node failure without problems. I also have a staging setup that can be converted to production quickly, and a terraform setup that can quickly spin up a Digital Ocean cluster if needed.

Re: Cloudflare had a partial outage

#293

Earlier quoted context omitted.

Probably not many users who need the performance and can handle unexpected failover. There would also be the issue of setting the policy defaults effectively. Most users wouldn’t benefit from this footgun. If you’re serious, you could probably automate this right now with your DNS provider and uptime monitoring.

I'm so serious that I already have failover after 1 hour at the registrar level, but those changes are not immediate and can take up to 24h to roll-in and roll-back due to DNS propagation and caching.

There is no immediate option with DNS changes. CF can’t immediately remove their IP from the route. Sounds like you’ve solved your problem in the sense that you have an automatic failover, though, which is good.

Re: Cloudflare had a partial outage

#294

Kind of ironic, there was a big "cloudflare is bad and a central point of failure" article on the front page just a couple days ago. Found it, https://news.ycombinator.com/item?id=31801947 edit: Not that I necessarily agree with the article even in light of there being an outage, cloudflare has been pretty good for us. Just thought it was interesting.

But web2 is going (really) great once you depend on Cloudflare. And it is certainly not re-centralizing the whole internet with a provider that is a single point of failure. /s

Re: Cloudflare had a partial outage

#295

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.

All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors. The Cloudflare Dashboard is also no longer fully loading.

Where are you located?

Re: Cloudflare had a partial outage

#297

Earlier quoted context omitted.

All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors. The Cloudflare Dashboard is also no longer fully loading.

Where are you located?

Raleigh, NC, USA

Sites are gradually reappearing as I type this. Some of my sites, and doordash.com, were returning 500 errors again just a minute ago. They just came back up, followed by the CF dashboard loading again.

Re: Cloudflare had a partial outage

#298
I really dislike that they are editing their status messages.

Entry[1] dated "Jun 21, 2022 - 06:43 UTC" has been edited to include more detail after they posted another entry at 06:57 UTC. There seems to be no indication that the message has been altered.

Currently text on the status page may suggest that they identified the problem immediately but it took about 15 minutes. Previously there was a text stating that customers should expect update within 15 minutes. Next message was posted 14 minutes after that but previous message was altered later and nothing indicates this.

Cloudflare, not cool.

[1] https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

Re: Cloudflare had a partial outage

#300

It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal). This is also important for countries with limited connectivity to the Internet, if the PoP in that c…

You can implement your own DNS server that CNAMEs to Cloudflare and falls back to origin IP when there is a problem with Cloudflare. I think a downstream Cloudflare provider could provide such services if they desire.

Outages occur to the best of CDNs. But the likelihood of two CDNs suffering an outage at the same time is close to zero. Check out www.cdnreserve.com.
Post reply on HN