Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

61–70 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#61

Post by yardstick »

I’ve asked this in a previous thread but didn’t get much of an answer: what can a business do to proactively fend off issues with cloud providers shutting them down.

I’ve heard GCP (or was it AWS?) is less likely to shut you down if you bill on account vs using a credit card.

Are there any other things we can do to reduce problems?

Would asking for an account manager help? Would buying reserved instances help?

Is there a minimum spend that gets you more attention and priority? Eg 100/month vs 1k/month vs 10k/month etc?

Any other advice?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#62

Post by bogomipz »

4ntonius8lock wrote: post #54
terryf wrote: post #32

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed. It was such a relief when times changed and regardless of who you were, you could start exchang…

Personally, I blame the decoupling of the dollar to the gold standard and the distribution of newly minted dollars from the federal reserve into the well connected via banks and corporations that are controlled by a small social segment that all attend the same schools. Once money became a thing that doesn't cost anything but the changing of zeros, then growth becomes a question not of how to produce something to get…

>"This has created money silos, where the US aristocracy will take hundreds of billions in loses to capture a market and then extract value in monopolistic ways."

I was curious about the above sentence. Who is the aristocracy in the context of startups? Are you referring to the VCs? If so don't the VCs not care about the "class" of the founders of a startup as long as they think there's money to be made from their company?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#63

Post by noir_lord »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

FWIW I do the same I have backups of everything.

All that said in close to a decade (actually might be a decade) I've still yet to have any issue with linode (tempting the computer gods here) or their backups.

I remember when DO was first starting out and they used to knock their own stuff offline updating router tables.

Ever since I have a jaundiced view of their capabilities.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#64

Post by freehunter »

barkingcat wrote: post #30
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because DO has their $5 and $10 / month plan, and that's really enough for me.

That's exactly it for me too. The Hetzner plan costs about twice as much as my monthly DO bill. Not to mention I'm US based and all of my customers are US based so hosting a site in Germany or Finland doesn't make a whole lot of sense.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#65

Post by jimnotgym »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

I'm moving Azure, AWS and other VPS workloads to them right now. Very impressive service!

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#66

Post by luckylion »

ehnto wrote: post #59
duxup wrote: post #36

Earlier quoted context omitted.

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…

I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does. I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones…

> Also, why would they want to?

One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money.

Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with you on proving your identity. Would you rather change your email on all your accounts or pay $50?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#67

Post by hadlock »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR.

Plus you get access to backups and all the other tools and products that are constantly being released. Not having to hire an IT guy at $1200/mo to maintain your physical server and tend to it's networking rules is a significant savings over a developer-managed aws/gcp instance. Plus all the cloud interfaces are standardized so you can just hire someone to come through and fix/upgrade your infrastructure.

Using a home-spun physical server means months if not years of undocumented tech debt for the next guy who comes along to have to maintain whatever kludges were installed long before he/she ever showed up. I just got done converting a bunch of physical servers over to the cloud and spent a year unwinding seven years of technical debt and now the dev/qa teams can actually spin up a new test environment in under 30 days (closer to 3 minutes).

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#68

Post by freehunter »

chocolatebunny wrote: post #40
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.

The reason for doing business with them is because they say they will deliver the service you paid them for. But if they don't, what then? You can switch to another vendor, but without a backup you're starting from scratch.

You should hope they deliver the service you've paid for, but you should never trust that they will. Always plan for failure. That's the entire point of this whole "DO killed my company" saga.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#69

Post by LinuxBender »

dmix wrote: post #17
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

These are strange times I guess? Maybe I'm a weird one-off, but I have 3 VPS providers for my hobby sites, my own DNS spread across all three (one remains shadow backup) and offline rsnapshot backups that automation can't touch.

I can't imagine doing less for a business. In my opinion, the business model and investment plan should account for all of this for at least the first five years.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#70

Post by turrini »

I've got a stable setup:

  ---------------------------------------
  Vultr ========================== Linode
  ---------------------------------------
  Los Angeles ==================== Dallas
  Mastercard 1234 ============= Visa 4567
  ---------------------------------------
  VMs with custom Linux (ROOT/ZFS Debian)
  Replicating snapshots every 5 minutes
  CloudFlare + HAProxy Load Balancing
Total cost: 6 VMS at $20 each: $120, CloudFlare $20

Edit: backups to B2 and Wasabi. ~$10 month

Post reply on HN