I use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.
Multiple Digital Ocean services down
31–40 of 46 posts
Re: Multiple Digital Ocean services down
#32I use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.
dig b1.eu-central-1.storage.railway.app +short
s3.eu-central-1.wasabisys.com.
eu-central-1.wasabisys.com.
Re: Multiple Digital Ocean services down
#33Earlier quoted context omitted.
slight off topic: I used DO LBs for a little while but found myself moving away from that toward a small droplet with haproxy or nginx setup. Worked much better for me personally!
The point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point. These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.
Re: Multiple Digital Ocean services down
#34Earlier quoted context omitted.
The point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point. These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.
If they are small projects, why are they behind a load balancer to begin with?
Re: Multiple Digital Ocean services down
#35Re: Multiple Digital Ocean services down
#36Earlier quoted context omitted.
The point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point. These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.
If they are small projects, why are they behind a load balancer to begin with?
Re: Multiple Digital Ocean services down
#37Earlier quoted context omitted.
If they are small projects, why are they behind a load balancer to begin with?
I use the LB's for high availability rather than needing load balancing. The LB + 2 web back-ends + Managed DB means a project is resilient to a single server failing, for relatively low devops effort and around $75/mo.
Did your high availability system survive this outage?
Re: Multiple Digital Ocean services down
#38Earlier quoted context omitted.
I use the LB's for high availability rather than needing load balancing. The LB + 2 web back-ends + Managed DB means a project is resilient to a single server failing, for relatively low devops effort and around $75/mo.
Are both servers deployed from the exact same repo/scripts? Or are they meaningful different, and/or balanced across multiple data centers? Did your high availability system survive this outage?
They wouldn't survive DO losing a DC, they're not so mission critical that it's worth the extra complexity to do that, and I don't recall DO losing a DC in the past 10 years or so.
They did stay up during this outage, which was apparently mostly concentrated on a different product called the 'global load balancer', which ironically is exactly the extra complexity I mentioned to in theory survive a DC outage.
Keep in mind these are "important" in the sense that they justify $100/mo on infra and monitoring, but not "life critical" in that an outage is gonna kill somebody or cost millions of bucks an hour. Once your traffic gets past a certain threshold, DO's costs don't scale that well and you're better off on a large distributed self-managed setup on Hetzner or buying into a stack like AWS.
To me their LB and DB products hit a real sweet spot -- better reliability than one box, and meaningfully less work than setting up a cluster with floating IP and heartbeats and all that for a very minimal price difference.
Re: Multiple Digital Ocean services down
#39So yesterday Azure got hit hard, today CF and DO are down, bad week or something else?