Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multiple Digital Ocean services down

status.digitalocean.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Multiple Digital Ocean services down

#21

Post by oasisbob »

potato3732842 wrote: post #8
matt-p wrote: post #6

Earlier quoted context omitted.

Just spaces CDN, not spaces - you'd think they'd just turn the CDN off for a bit.

You can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.

Might be just me, but I can think of many origins under my control which could live without a (non-functional) CDN for a while.

CDN is great for peak-load, latency reductions, and cost - but not all sites depend on it for scale 24/7

Re: Multiple Digital Ocean services down

#23

Post by giancarlostoro »

matt-p wrote: post #15
giancarlostoro wrote: post #9

So yesterday Azure got hit hard, today CF and DO are down, bad week or something else?

DDOS, but I don't really understand why in particular.

Having known people like this, its either flexing about who has the more powerful botnet or advertising who can do what.

Re: Multiple Digital Ocean services down

#24

Post by miken123 »

showerst wrote: post #10

I use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.

It is mentioned in their list of subprocessors: https://www.digitalocean.com/trust/subprocessors

Re: Multiple Digital Ocean services down

#25

Post by graemep »

matt-p wrote: post #12
potato3732842 wrote: post #8

Earlier quoted context omitted.

You can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.

If you are DO you could, you just decided not to bother. They control the origins it's spaces (s3), so they could absolutely spin up further gateways or a cache layer and then turn the CDN off.

Either you are wrong and they do not have the capacity to do that, or they have decided it is acceptable to be down because a major provider is down

I imagine a cache layer cannot be that easy to spin up - otherwise why would they outsource it?

Re: Multiple Digital Ocean services down

#26

Post by matt-p »

graemep wrote: post #25
matt-p wrote: post #12

Earlier quoted context omitted.

If you are DO you could, you just decided not to bother. They control the origins it's spaces (s3), so they could absolutely spin up further gateways or a cache layer and then turn the CDN off.

Either you are wrong and they do not have the capacity to do that, or they have decided it is acceptable to be down because a major provider is down I imagine a cache layer cannot be that easy to spin up - otherwise why would they outsource it?

You outsource it because clouflare have more locations than you so offer lower latency and can offer it at a cost that's cheaper or the same price as doing it yourself.

Re: Multiple Digital Ocean services down

#27

Post by graemep »

matt-p wrote: post #26
graemep wrote: post #25

Earlier quoted context omitted.

Either you are wrong and they do not have the capacity to do that, or they have decided it is acceptable to be down because a major provider is down I imagine a cache layer cannot be that easy to spin up - otherwise why would they outsource it?

You outsource it because clouflare have more locations than you so offer lower latency and can offer it at a cost that's cheaper or the same price as doing it yourself.

Which suggests its expensive enough for it to be unlikely they just have the capacity lying around to spin up.

Re: Multiple Digital Ocean services down

#28

Post by showerst »

wesammikhail wrote: post #20
showerst wrote: post #10

I use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.

slight off topic: I used DO LBs for a little while but found myself moving away from that toward a small droplet with haproxy or nginx setup. Worked much better for me personally!

The point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point.

These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.

Re: Multiple Digital Ocean services down

#29

Post by tgma »

potato3732842 wrote: post #8
matt-p wrote: post #6

Earlier quoted context omitted.

Just spaces CDN, not spaces - you'd think they'd just turn the CDN off for a bit.

You can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.

nit: that's more DoS (from a handful of DO LBs) than DDoS.

Re: Multiple Digital Ocean services down

#30

Post by fridder »

Lammy wrote: post #18
zx8080 wrote: post #13

Earlier quoted context omitted.

Year-end promotion cycle is the worst time for end-users and the best one for engineers greedy for promotions.

Don't blame individual engineers who want to do what will be rewarded instead of company performance policies that reward this type of behavior.

shoot, there are also end of year layoffs and reorgs to pump up those year end numbers
Post reply on HN