Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

71–80 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#71
post #45

Earlier quoted context omitted.

Why is everyone talking about backups and not the actual infrastructure?

because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.

They did have backups, only problem backups was also on DO. Backing up petabyte of data even on a Gbit connection is slow, and also cost more then a pennies if you for example back it up on AWS. Linus tech tips have a video on youtube explaining the problem.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#72
post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

This isn't a DigitalOcean option (I don't think anyways, I've never seen it), but on AWS/GCloud/Azure you can absolutely exchange money in order to get faster support SLAs, or even AWS solutions experts hanging out in your slack channel.

Of course, people who don't have money are forced to deal with greater hardships, yet that's always been a footnote to "regardless of who you were, you could start exchanging money".

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#73

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Do you not have a mirror copy of everything somewhere else? Or is that not possible in your case?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#74
post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because DO has their $5 and $10 / month plan, and that's really enough for me.

https://aws.amazon.com/lightsail/pricing/?opdp1=pricing ?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#75

I'm a one-man startup and I couldn't disagree with this more. If you want to "play" startup, feel free to put all your eggs in one basket. If you actually want to build something meaningful and long-lasting, and you have Fortune 500 customers, take a breath and invest time in an infrastructure that can't be destroyed by a single rogue algorithm. After all of the horror stories we've heard over the years, how is that…

> take a breath and invest time in an infrastructure that can't be destroyed by a single rogue algorithm

As in? You can have your own cage, but somebody will have a cable that connects to the hardware you own. If they pull that cable because a single rogue algorithm told them that you're most likely abusing their service, that's it.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#76
post #17

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

Local backups? A few hard drives and a fiber connection can be cheap. Slow to restore, but not "killed my business" slow

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#77
post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

Yeah, support by social media seems to be trend now; those who are loudest and have the largest online followings get support by shouting at companies on Twitter, everyone else is just basically told to pound sand.

Then again, it's not just support for services. Even abuse reports seem to be taken a lot more seriously it's some online 'influencer' at the receiving end rather than an average Joe.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#78

I’ve asked this in a previous thread but didn’t get much of an answer: what can a business do to proactively fend off issues with cloud providers shutting them down. I’ve heard GCP (or was it AWS?) is less likely to shut you down if you bill on account vs using a credit card. Are there any other things we can do to reduce problems? Would asking for an account manager help? Would buying reserved instances help? Is the…

- Use billing methods which are less likely to be used by scammers (e.g. likely not credit card).

- Keep a steady balance.

- Have luck (sad but true).

Minimum spend and similar might help with customer service (if they can see your spend). But most problems start with automated services.

Don't switch cloud providers every half year because another one is slightly cheaper now (i.e. have longer running/existing accounts), might help.

But most important make sure that you don't get locked in by the could provider

Even a permanent ban from the could provider should just cause some, not to high, short term (mony) loss. This means:

- Do not ~use~ relay on cloud providers proprietary tec (sure use there management console, etc. but don't integrate it into your business to tightly).

- Have control over your DNS names so that you can map them to different IP addresses if needed.

- Make sure you can migrate to a different cloud provider in matters of hours/days/weeks (depending on what you use it for) at any time.

- Have backups outside of the could provider for all your stuff. - don't get trapped by doing backups on a different service of the same provider (e.g. Amazon) ;=)

In the end how much this costs you depend a lot on what you do and how you do it. And yes deciding and not doing any continous shot term investments taking the risk of loading everything is fine, you just should make the decision conscious.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#79
post #36
post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…

>I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role.

The crazy thing is that it's really not that expensive. AWS, for example, offers 24x7 support w/ <1 hour response time for production issues starting at $100 a month. At worst it's 10% of your bill.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#80
post #70

I've got a stable setup: --------------------------------------- Vultr ========================== Linode --------------------------------------- Los Angeles ==================== Dallas Mastercard 1234 ============= Visa 4567 --------------------------------------- VMs with custom Linux (ROOT/ZFS Debian) Replicating snapshots every 5 minutes CloudFlare + HAProxy Load Balancing Total cost: 6 VMS at $20 each: $120, Clou…

Add a local backup. Then again, Im biased :D
Post reply on HN