If you're reading this and concerned for your own backup story, fret not! In 2019 secure off-premise backups are super easy to implement, even for a 1 person shop. Get something like Restic or Borg or any one of the enumerated options here: https://github.com/restic/others I've recently implemented backups with Restic, the static binary and plethora of supported storage backends was extremely appealing. The easiest s…
"DigitalOcean Killed Our Company"
231–240 of 620 posts
Re: "DigitalOcean Killed Our Company"
#232Re: "DigitalOcean Killed Our Company"
#233Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…
Re: "DigitalOcean Killed Our Company"
#234Earlier quoted context omitted.
More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
>getting locked out of your account is usually just not a consideration How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan. Dumping everything to B2 is trivially easy, triv…
Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
Re: "DigitalOcean Killed Our Company"
#235Earlier quoted context omitted.
The ability to adapt is the definition of a contingency plan. It's essentially, "If this person/service/database/customer/etc vanished off the face of the earth, what do?"
Okay, then that means they did have a contingency plan, except for a single rsync. Which would mean you disagree strongly with coldtea?
Their entire business was completely reliant on DO droplets. It doesn't take much foresight to think, "hey, I should probably make a backup in case this VPS goes down."
Nothing in this comment thread, or the OP twitter thread mentions anything about the rest of this imaginary contingency plan of theirs.
Re: "DigitalOcean Killed Our Company"
#236Earlier quoted context omitted.
This race to the bottom has reached a point that it's harming customers. It's okay to be more expensive than the competition if you provide a better service.
Personal opinion, it's really important in the ISP/hosting world to identify what market categories are a race to the bottom, and if at all possible, refuse to participate in them. I look at companies selling $5 to $15/month VPS services and try to figure out how many customers they need to be set up for monthly recurring services, in order to pay for reasonably reliable and redundant infrastructure, and the math jus…
Re: "DigitalOcean Killed Our Company"
#237Hopefully everything gets righted and these folks can start making solid, multi-site backups.
Re: "DigitalOcean Killed Our Company"
#238Earlier quoted context omitted.
> That's simply not true. There's support engineers hired around the world, and depending on when your ticket is posted, someone awake at that time will answer. DO is super remote friendly and as a result, has employees (and support folks) everywhere on the planet. Not "cheap oversea support" at all. There's a lot of support folks in the US, Canada, Europe, and in India where they have a datacenter. Going to guess fr…
I don't work at DO anymore, but you're right I should disclose that I used to do so. Will edit my original post.
Re: "DigitalOcean Killed Our Company"
#239Earlier quoted context omitted.
It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.
More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
I could not disagree more. There's a right way and a wrong way to do this, it's trivial to do it right, and the risks of doing it wrong are enormous.
Re: "DigitalOcean Killed Our Company"
#240Earlier quoted context omitted.
This is conflating two different things. One point is valid, the other is not. - No offsite backups? Agreed. Even for a two person team it is sloppy. - "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinkin…
Some may have an availability dependency on those services, but if they don't have a full BC and DR plan ready to go within a few hours of losing those service they're not going to be a big enterprise for long.
How you got from "single tech partner" to "have no disaster recovery plan" I don't know.