Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

21–30 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#21

Post by EdwardDiego »

scarface74 wrote: post #19
EdwardDiego wrote: post #11

Earlier quoted context omitted.

> So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource? The account that got locked down triggered automatic checks that used payment history as a "these people are okay" check. They didn't have a payment history, they were running solely on credits, so got flagged. IOW, they…

You have 24 hours to respond and why would you have a business and not be on at least their business support plan?

That's a very good question for their CTO. Maybe we should ask him on Twitter.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#22

Post by jwr »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance

I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting.

For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better reliability, rent three.

I looked at this many times and still check the pricing regularly, and it just doesn't make sense for me to switch from these physical servers to virtualized offerings, not in any foreseeable future.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#23

Post by scarface74 »

I do not have all the details, so caveat lector, but apparently it was due to spinning up a dozen droplets and running a batch job.

If this is considered “abuse” Digital Ocean is not a platform you should be running a business on.

If you can’t afford to pay for reliable big boy infrastructure, you might need to rethink your business plan or how much you charge.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#24

Post by crazygringo »

cm2187 wrote: post #3
ianlevesque wrote: post #2

> Do have some backups outside of your primary cloud provider. You'll sleep better. This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).

And have a separate DNS provider.

No reason to.

If your cloud provider does your DNS and you switch to a different cloud provider...

...then just set your registrar to point to the new DNS provider. You've still got total control.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#25

Post by mabbo »

It feels like the need for multi-cloud designs and infrastructure are becoming more sensible. If costs are best with one provider, by all means put 95% of your traffic there... But keeping 5% running smoothly on a second cloud, with a simple means to rebalance between them would be a huge win. It would mean the difference between an annoyance (argh, one of my cloud providers doesn't work) and a disaster (my sole cloud provider just ended my company). It would also save you the next time AWS or Google's cloud have an outage.

Are there any good resources for deploying multi cloud architectures?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#26

Post by tinus_hn »

cm2187 wrote: post #3
ianlevesque wrote: post #2

> Do have some backups outside of your primary cloud provider. You'll sleep better. This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).

And have a separate DNS provider.

Absolutely, your domain name might be your most important asset. It’s silly to put that in the hands of a party that you might have to fight. Especially as there are hundreds of alternatives.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#27

Post by geggam »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because I can simply turn off everything and reduce my spend to storage alone. If I turn off that server I still pay for it

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#28

Post by pbourke »

dmix wrote: post #17
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

But you don’t need the same architecture replicated across two providers, as your parent post demonstrated. Run Postgres in DO and upload your backups to S3/Azure Storage. This is both cheap and simple to the level that any startup could manage it. If and when it makes sense, you can investigate multi provider replication, but it might just be YAGNI.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#30

Post by barkingcat »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because DO has their $5 and $10 / month plan, and that's really enough for me.
Post reply on HN