Why the “Digital Ocean killed my company” incident scares the hell out of me • Hacker News Board

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#91

Post by phoobahr » Thu, Jun 06, 2019, 6:59 PM UTC

freehunter wrote: post #64

barkingcat wrote: post #30

Earlier quoted context omitted.

Because DO has their $5 and $10 / month plan, and that's really enough for me.

That's exactly it for me too. The Hetzner plan costs about twice as much as my monthly DO bill. Not to mention I'm US based and all of my customers are US based so hosting a site in Germany or Finland doesn't make a whole lot of sense.

And here I am a Canadian and my client base is Canadian. While not a problem just yet it is likely that PIPEDA/PHIPA would make it complicated to serve data from outside the country.

I am not a lawyer but the advice I have been given is murky. It’s much easier to stuck with DO which, at my usage, costs about the same number of $CAD as lightsail would in $USD

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#92

Post by DKnoll » Thu, Jun 06, 2019, 6:59 PM UTC

fhood wrote: post #51

Arguably Raisup should have been better prepared for this eventuality. But , I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before. I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get h…

> Or maybe the guy was an idiot. But possibly entertain the thought that maybe his situation wasn't exactly comparable to your eh?

I have never encountered a business that does not require backups. Not having a DR site as early in the game as they were might be excusable, but even then suggesting they get one is sound advice. It's an additional expense but it has a cheaper implementation cost while your infrastructure is small. Having a good DR plan is a selling point as well, this is the first thing I ask SaaS providers when I am evaluating them.

What is a total failure is not having backups on another provider. Before anyone says it's just a 2-person startup... surely that makes it easier? I can't imagine their DBs are all that big. Even just a three line cronjob and a Backblaze B2 ($0.005/GB) account could have secured their business continuity in the event their DO account didn't come back up.

>But also DO Spaces for our object storage with ~500k media and our database backups.

Based on that comment (from @w3nicolas on Twitter) they actually could have used the same script they used for DO Spaces to back up their DBs to S3 (both use the S3 protocol). The 500k image files should be relatively small and cheap to back up too, appears they are just logo thumbnails for the tracked companies (500k image files, copy says they track 450k companies).

If you are building infrastructure and you don't have IT Ops experience consult somebody who does because they will save you from yourself in scenarios like this. If you are a dev you could probably throw a rock and hit 5 friends with enough ops experience to tell you the importance of redundant backups.

I think this comment might read a bit more abrasively than I meant it, but I don't mean to kick the guy after he was down. If you are in a similar boat to RaisUp with your backups go fix it now. If you rely on any one vendor for business continuity that is a problem. This PSA is sponsored by salty sysadmins everywhere.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#93

Post by Alex3917 » Thu, Jun 06, 2019, 7:00 PM UTC

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But it wasn't, because every night I run a PG backup and copy it to AWS S3

Which is probably much easy on DO than AWS. For whatever reason AWS seems to go out of their way to make it as hard as possible to backup your RDS data to a separate AWS account.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#94

Post by pkaye » Thu, Jun 06, 2019, 7:00 PM UTC

luckylion wrote: post #66

ehnto wrote: post #59

Earlier quoted context omitted.

I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does. I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones…

> Also, why would they want to? One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money. Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with…

There is Google Apps which is a paid service but most people gravitate to the free stuff.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#95

Post by adolph » Thu, Jun 06, 2019, 7:01 PM UTC

phonon wrote: post #74

barkingcat wrote: post #30

Earlier quoted context omitted.

Because DO has their $5 and $10 / month plan, and that's really enough for me.

https://aws.amazon.com/lightsail/pricing/?opdp1=pricing ?

Nice, use Docker containers in it:

https://www.youtube.com/watch?v=z525kfneC6E

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#96

Post by rkangel » Thu, Jun 06, 2019, 7:13 PM UTC

jwr wrote: post #22

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

In our case, because it was easy to get thousands of dollars of free credit in Google cloud (the same appeared to be true for AWS). This went up to tens of thousands for Series A.

We had zero serving costs and would have continued to do so for years.

It makes perfect sense for Google/Amazon to incentivise startups to design themselves for their cloud.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#97

Post by cm2187 » Thu, Jun 06, 2019, 7:14 PM UTC

crazygringo wrote: post #24

cm2187 wrote: post #3

Earlier quoted context omitted.

And have a separate DNS provider.

No reason to. If your cloud provider does your DNS and you switch to a different cloud provider... ...then just set your registrar to point to the new DNS provider. You've still got total control.

That transition requires authorisation from the original provider. If they locked your account and don't pick up the phone, you won't get the move approved and won't be able to repoint your urls to a different cloud.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#98

Post by hadlock » Thu, Jun 06, 2019, 7:17 PM UTC

social_quotient wrote: post #88

hadlock wrote: post #67

Earlier quoted context omitted.

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR. Plus you get access to backups and all the other tools and…

$200/mm for a 10MM ARR company. Can you add more info here. What sort of app/company are we taking about here? Genuinely curious

For production, $200/mo. Dev/testing seem to always cost 3x+ what production servers cost.

One was an IT management/automation SAAS company (I think they were $450/mo), another was backoffice SAAS company (I think they were $1000/mo but they were also hosting for an independent dealer of their SAAS so it was two production systems, and 10 years of tech debt), third is a live video consultancy SAAS that outsourced their video to a third party api service. None of those are actually $200/mo but realistically in the ballpark of under $1000/mo.

When I converted the company (which included 40 engineers and probably 12 qa engineers) to the cloud I got my hands slapped for converting the QA/test datacenter (not production) and the cost went from $3200/mo on bare metal to $6500/mo but we were able to squeeze that down to $4200/mo running about 10 always-on test/pre-prod environments and 2-3 on-demand test environments.

This is all enterprise space stuff where the utilization is low but the value of the service is high. If you're doing facebook for dogs and trying to turn a profit on ad revenue from high utilization it's probably not as effective/realistic.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#99

Post by 4ntonius8lock » Thu, Jun 06, 2019, 7:18 PM UTC

bogomipz wrote: post #62

4ntonius8lock wrote: post #54

Earlier quoted context omitted.

Personally, I blame the decoupling of the dollar to the gold standard and the distribution of newly minted dollars from the federal reserve into the well connected via banks and corporations that are controlled by a small social segment that all attend the same schools. Once money became a thing that doesn't cost anything but the changing of zeros, then growth becomes a question not of how to produce something to get…

>"This has created money silos, where the US aristocracy will take hundreds of billions in loses to capture a market and then extract value in monopolistic ways." I was curious about the above sentence. Who is the aristocracy in the context of startups? Are you referring to the VCs? If so don't the VCs not care about the "class" of the founders of a startup as long as they think there's money to be made from their co…

You are right, they don't care. If they did, someone would take their place. It's a system, not a conspiracy.

But an Ivy graduated person is a class of person that is highly committed to the system. There's a massive effort a whole family has to make for someone to be there.

If success is based on capturing the most market share the quickest so you can corner the market and extract value via monopolistic methods, then why go with anyone who could be a risk?

Edit: Re who is the aristocracy? There is no exact line. Society works on a gradient of privilege, from those who have access to resources with the least effort, to those who have access to resources with the most effort. This always happens. But I believe it is aggravated when money doesn't cost anything to produce because risk is then more important than reward.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#100

Post by duxup » Thu, Jun 06, 2019, 7:21 PM UTC

treis wrote: post #79

duxup wrote: post #36

Earlier quoted context omitted.

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…

>I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. The crazy thing is that it's really not that expensive. AWS, for example, offers 24x7 support w/ <1 hour response time for production issues starting at $100 a month. At worst it's 10% of your bill.

It's very doable, but sadly even skilled management (in my experience) tends to avoid / flee support organizations due to their low prestige / resources pattern that is across a lot of industries.

I fear it is something very doable, but I'm not sure there are many in leadership that can.