Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

11–20 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#11

Post by EdwardDiego »

chrismeller wrote: post #4

> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource? I think this is something that needs…

> So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource?

The account that got locked down triggered automatic checks that used payment history as a "these people are okay" check. They didn't have a payment history, they were running solely on credits, so got flagged. IOW, they hadn't paid for anything yet.

> With AWS I can spin up as many “boxes” as I want on a moment’s notice

You can, but they can also shut you down for "abuse". https://aws.amazon.com/premiumsupport/knowledge-center/aws-a...

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#12

Post by freehunter »

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch.

But it wasn't, because every night I run a PG backup and copy it to AWS S3. I just had to download the backup from the other cloud vendor and restore it on my DO server.

Did DO fuck up? Yes. Did it cost me downtime? Yes. Was I mad? Yes, and I still am. But I still do business with DO because it costs ~half the price of a comparable EC2 instance, and writing a 10 line bash script to move my database backups to another cloud vendor isn't that hard. Storing that backup on S3 costs pennies per month.

I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. When it comes down to it, you are solely responsible to your customers. They're not going to care whose fault it was, because it was your fault.

Don't trust any of your vendors.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#13

Post by duxup »

Digital Ocean's statement on the events:

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

Lots of support related issues here.

As someone who worked in support for a long time it isn't surprising to see this play out and the whole "Support and Security Operations leadership will create new workflows to allow abuse-related events to leverage the 24/7 structure of Support." while probably a good action to take, is one of those things that you see in support time and again and so rarely do you see an appropriate response as much as a patch for "this one thing won't happen again".

Nobody cares to staff, fund, and support the support teams until something goes wrong, and then it is usually "new workflows" for support.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#14

Post by corobo »

DenisM wrote: post #10
cm2187 wrote: post #3

Earlier quoted context omitted.

And have a separate DNS provider.

Does that really help? If your DNS provider screws it up, you're in deep trouble regardless of whether they were the same guys hosting your VMs or not.

That's when you go to your registrar (you do have that separated off too, right?) and change your nameservers

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#15

Post by chrisan »

DenisM wrote: post #10
cm2187 wrote: post #3

Earlier quoted context omitted.

And have a separate DNS provider.

Does that really help? If your DNS provider screws it up, you're in deep trouble regardless of whether they were the same guys hosting your VMs or not.

In the original article this is referencing DO shut down/locked his account.

By having an alternate provider you spread your risk. And in this scenario you could update your DNS from the locked DO account to AWS/Linode/GCP whatevr

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#16

Post by reading-at-work »

I recently tried to sign up for a DO account to deploy a proof of concept app for my startup, and my account was immediately locked within minutes of opening it. I hadn't even spun up an instance yet. That combined with this means I'll be sticking with AWS for now, I suppose.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#17

Post by dmix »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section.

> - Why were you only hosting one JUST ONE cloud provider?

> - Why didn't you have backups outside of JUST ONE cloud provider?

Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not to have extended downtime.

This is much bigger issue than having external backups (which every startup should still do).

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#18

Post by mark-ruwt »

I'm a one-man startup and I couldn't disagree with this more.

If you want to "play" startup, feel free to put all your eggs in one basket. If you actually want to build something meaningful and long-lasting, and you have Fortune 500 customers, take a breath and invest time in an infrastructure that can't be destroyed by a single rogue algorithm.

After all of the horror stories we've heard over the years, how is that time investment (and ability to sleep well at night) not worth it?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#19

Post by scarface74 »

EdwardDiego wrote: post #11
chrismeller wrote: post #4

> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource? I think this is something that needs…

> So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource? The account that got locked down triggered automatic checks that used payment history as a "these people are okay" check. They didn't have a payment history, they were running solely on credits, so got flagged. IOW, they…

You have 24 hours to respond and why would you have a business and not be on at least their business support plan?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#20

Post by zamu »

dmix wrote: post #17
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.
Post reply on HN