Why the “Digital Ocean killed my company” incident scares the hell out of me
1–10 of 185 posts
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#2> Do have some backups outside of your primary cloud provider. You'll sleep better.
This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#3> Do have some backups outside of your primary cloud provider. You'll sleep better. This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).
And have a separate DNS provider.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#4> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors).
So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource?
I think this is something that needs to be much more front and center. With AWS I can spin up as many “boxes” as I want on a moment’s notice. As far as anything on DO’s site seems to advertise I can do the same.
Turns out that’s not so. If they wanted to shut down an account because they suspected it was compromised then that’s one thing. Shutting it down simply because of suspected cryptocurrency activity... not so much. If I’m willing to pay for it then that’s what I’m willing to pay for and there should be no limitations.
Now that becomes a little more clear as you read...
> determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers.
So clearly we are not talking the same resources that were expected.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#5Any stories of being screwed by AWS in the similar manner?
That is, excluding the stories about the same account being used to sell thing on Amazon.coma and run a AWS setup. We know those are risky.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#6> Why isn't everyone on bare metal?
Is this how we're referring to running on servers we have contractual control over these days? Part of managing risk is being sure your infrastructure will stay there, and if you don't have a contract, your risk is significantly higher.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#7Pretty much the standard "the cloud is just someone else's computer" issue: if you don't have the ability to reproduce your work somewhere else and your provider decides to go away, you're going to have a Bad Time(TM).
Also, for as cool as Digital Ocean is, their primary focus is on low-use, shared cloud resources. From my experience, they over subscribe CPU resources so "noisy neighbors" was a problem sometimes. They do not provide or work with people as if they have production services, and they don't seem to like people who really want to use their system. I would never use them for production unless the work was ephemeral.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#8My account was banned two years ago just for accessing it from abroad. This is real, don't use DO.
They finally gave me temporary access even though they didn't believe who I was. This was the icing on the cake, they gave me access to a cluster, they could've given it to anyone.
It's cheap but seriously not worth it. They also have many more outages than other larger providers.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#9You don't have to actually have redundant providers and monthly-tested DR runs, etc. But you do need a plan. No matter how small.
If you are just a side project running locally, than plan me be as simple as "If my laptop falls in a river, get a new one and clone the repo again." As you start having production systems, it may still be "clone the repo again", but this time to actual servers.
As you start having customers rely on you, your plan should get more robust. Maybe you don't have a hot DR site set up where you can just flip a switch. But you should know who your backup host would be, and have an account ready with them. You should what steps would be needed to go from repo -> new provider.
None of this needs to be set up and tested ahead of time if you are just getting started. But if you have paying customers, you have to have thought about it. DR starts tiny and scales up, just like everything else.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#10> Do have some backups outside of your primary cloud provider. You'll sleep better. This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).
And have a separate DNS provider.
Does that really help? If your DNS provider screws it up, you're in deep trouble regardless of whether they were the same guys hosting your VMs or not.