Uh, that’s not good. The negatives of centralisation really smack you in the face.
I’ll start moving my sites away from Cloudflare soon. Not because it’s bad — in fact it has been amazing, but rather to decentralise.
Cloudflare had a partial outage
231–240 of 448 posts
Re: Cloudflare had a partial outage
#232Quoted post unavailable.
No post body was provided.
Re: Cloudflare had a partial outage
#233Re: Cloudflare had a partial outage
#234Looks like it's mostly good now?
Re: Cloudflare had a partial outage
#235If only we could come up with a globally distributed set of networks and systems that could be run by millions of entities that don't rely on each other to keep working. Oh no wait...
Re: Cloudflare had a partial outage
#236Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.
hats off to you sir - would not want to be in your shoes right now but thanks for the updates
I don't have shoes on.
Re: Cloudflare had a partial outage
#237Am I wrong to use this as an excuse to not use Cloudflare?
Apparently yes, people in here are quick to defend our new overlord.
"But they're great", they cry, "why should I use anything else?"
Re: Cloudflare had a partial outage
#238Our monitoring systems had influxdb endpoints behind cloudflare. Now we not only lost users but also access to data about the impact of outage.
Re: Cloudflare had a partial outage
#239At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.
What is your setup to where you are isolated from "other person making a mistake"? Even if you're a box in a colocated datacenter you're still able to get knocked off the net from some maintenance on the surrounding pipes. Hell, hosting your own box doesn't cause Comcast DNS issues to not knock off a bunch of people either. I do think there is some holistic overview of hosting stuff on the internet, where you could l…
If you need fault tolerance/isolation, you want to have a second box in a different colo (preferably in a different city; a different coast/continent if it's important).
If you can live with dns round robin between the two, then you can easily host the DNS with multiple providers and avoid SPOF (could maybe host it on the two boxes you already have, too). You're still at risk of domain registry/registrar failures, and failures of their tld nameservers (very rare for well run tlds) and the root servers (not sure if they ever had a widespread failure). And of course, simultaneous failure of both locations isn't impossible, just less likely.
On Comcast DNS failures... Most of the recent ones I've heard of manifested as users on Comcast can't resolve X, but were really X had bad DNSSEC records and Comcast DNS refused to return records that weren't signed properly. It's easy to avoid that by not using DNSSEC.
In the general case of working despite bad ISP dns, you can't do much (anything?) for web browsers, but if you build apps, you can hard code fallback IPs for when DNS doesn't work... But you need to have IPs that stick around for the lifetime of your app downloads.
Re: Cloudflare had a partial outage
#240maybe we should start suggesting the use of IPFS/Arweave to host critical infra status pages or similar sites that are occasionally imminently needed