Live data from Hacker News

Cloudflare Global Network experiencing issues

cloudflarestatus.com

771–780 of 1001 posts

Re: Cloudflare Global Network experiencing issues

#774

I sometimes question my business decision to have a multi-cloud, multi-region web presence where it is totally acceptable to be down with the big boys.

Many services have just disabled the CF proxy and use only DNS. If your end server has SSL and can handle some traffic, it might work for a while.

Re: Cloudflare Global Network experiencing issues

#775

Speaking of 5 9s, how would you achieve 5 9s for a basic CRUD app that doesn't need to scale, but still be globally accessible? No auth, micro services, email or 3rd party services. Just a classic backend connected to a db (any db tech, hosted wherever), that serves up some html.

Part of the up-time solution is keeping as much of your app and infrastructure within your control, rather than being at the behest of mega-providers as we've witnessed in the past month: Cloudflare, and AWS.

Probably:

- a couple of tower servers, running Linux or FreeBSD, backed up by a UPS and an auto-run generator with 24 hours worth of diesel (depending on where you are, and the local areas propensity for natural disasters - maybe 72 hours),

- Caddy for a reverse proxy, Apache for the web server, PostgreSQL for the database;

- behind a router with sensible security settings, that also can load-balance between the two servers (for availability rather than scaling);

- on static WAN IPs,

- with dual redundant (different ISPs/network provider) WAN connections,

- a regular and strictly followed patch and hardware maintenance cycle,

- located in an area resistant to wildfire, civil unrest, and riverine or coastal flooding.

I'd say that'd get you close to five 9s (no more than ~5 minutes downtime per year), though I'd pretty much guarantee five 9s (maybe even six 9s - no more than 32 seconds downtime per year) if the two machines were physically separated from each other by a few hundred kilometres, each with their own supporting infrastructure above, sans the load balancing (see below), through two separate network routes.

Load balancing would become human-driven in this 'physically separate' example (cheaper, less complex): if your-site-1.com fails, simply re-point your browser to your-site-2.com which routes to the other redundant server on a different network.

The hard part now will be picking network providers that don't use the same pipes/cables, i.e. they both use Cloudflare, or AWS...

Keep the WAN IPs written down in case DNS fails.

PostgreSQL can do master-master replication, but it's a pain to set up I understand.

Re: Cloudflare Global Network experiencing issues

#776

How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.

This might sound crazy as a software engineer, but I actually like the occasional "snow day" where everything goes down. It's healthy for us to all disconnect from the internet for a bit. The centralization unintentionally helps facilitate that. At least, that's my glass half full perspective.

Re: Cloudflare Global Network experiencing issues

#777

There's something maliciously satisfying about seeing your own self-hosted stuff working while things behind Cloudflare or AWS are broken. Sure, they have like four more nines that me, but right now I'm sitting pretty.

My selfhosted sites are down because of the cloudflares proxy. Ugh

Re: Cloudflare Global Network experiencing issues

#780

AWS, Azure, now Cloudflare, all within a month, are hit with configuration errors that are definitely neither signs of more surveillance gear being added by government agencies nor attacks by hostile powers. It's a shame that these fine services that everyone apparently needs and that worked so well for so long without a problem suddenly all have problems at the same time.

Most or all of these lost significant institutional knowledge through layoff after layoff and jobs moved to lower cost countries.

Maybe a coincidence or maybe not.

Post reply on HN