Why the “Digital Ocean killed my company” incident scares the hell out of me • Hacker News Board

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#31

Post by Someone1234 » Thu, Jun 06, 2019, 5:21 PM UTC

jwr wrote: post #22

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective.

That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since you have to set up the logic of when to failover.

Auto-scaling is a benefit of "cloud" services. But it isn't the only selling point. Hardware abstraction is perhaps bigger.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#32

Post by terryf » Thu, Jun 06, 2019, 5:21 PM UTC

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed.

It was such a relief when times changed and regardless of who you were, you could start exchanging money for things and services.

This story and many others like it remind me of those times - it's back to who you know to get your account unlocked. I've never gotten a story on the front page of HN and probably have 5 twitter followers. What's my avenue of getting my data back?

This isn't just about DO either. Similar stories about google and other services are many to be found.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#33

Post by EugeneOZ » Thu, Jun 06, 2019, 5:23 PM UTC

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

It's because price of your site downtime is insignificant. DO (and many other "bulk vps" services) is only suitable for projects like this. When 1 hour of downtime cost much more than year of hosting, you will be ready to pay for high availability and duplication, replication, backups verification.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#34

Post by Sawamara » Thu, Jun 06, 2019, 5:25 PM UTC

There is a heck of a difference between "eh, it only costs $5-$10" and between "the survival of my project relies on the site/service being under my control. If it was the latter, you go in having a plan B and a contingency Plan c.

GCP, AWS, DO, Azure... they all bank on that sweet combination of huge clients and clients who do not really need them but due to opportunity costs, cant be bothered to find viable, self-reliant alternatives.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#35

Post by INTPenis » Thu, Jun 06, 2019, 5:26 PM UTC

Something tells me it scares DO too because I've been getting emails lately asking if I'm happy with their services.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#36

Post by duxup » Thu, Jun 06, 2019, 5:26 PM UTC

terryf wrote: post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role.

They want to automate that all away as much as possible.

The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy talks about how he was banned on X and is on Z now... simply for that reason.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#37

Post by EugeneOZ » Thu, Jun 06, 2019, 5:26 PM UTC

jwr wrote: post #22

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Maybe because they had dozens of failures and DDOS outages? Why they have to keep their prices so low - because of reputation. Price is the only attractive thing in services like this.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#38

Post by codegeek » Thu, Jun 06, 2019, 5:27 PM UTC

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Great point. I always run a separate backup through bash scripts that syncs with S3 bucket even though I also have the DO backups enabled as well. You always want 2 separate offsite backups for any real world production application.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#39

Post by EugeneOZ » Thu, Jun 06, 2019, 5:28 PM UTC

dmix wrote: post #17

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

No, it's not so difficult task and not even expensive. I don't think every project really needs it, though. But external db backups - they are ridiculously cheap and every project needs them.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#40

Post by chocolatebunny » Thu, Jun 06, 2019, 5:28 PM UTC

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.