Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

131–140 of 620 posts

Re: "DigitalOcean Killed Our Company"

#131

Post by tssva »

BinaryIdiot wrote: post #85
jlv2 wrote: post #40

Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."

Maybe. But don't forget this is a small company of just two people. Yes the backups should have been off site but relying on just one digital partner at that scale isn't the worst (they're unlikely to have the money or time to federate out to other services). Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on a…

I disagree. Many 1 and 2 person shops implement proper off-site backups because they understand that loosing their data is a death sentence for their business. Proper off-site backups are neither expensive nor time consuming to establish these days. There is no good excuse for even the tiniest of companies to not implement them.

Re: "DigitalOcean Killed Our Company"

#132

Post by hyperpape »

To everyone justifying the lack of a backup strategy by saying they’re a two man show:

As far as sympathy goes, you’re not wrong. But you’re also justifying every pain in the ass procurement process you’ve ever dealt with. Your attitude is why so many companies won’t go near a two man shop.

Re: "DigitalOcean Killed Our Company"

#133

Post by _bxg1 »

stingraycharles wrote: post #122
_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup.

Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.

Re: "DigitalOcean Killed Our Company"

#134

Post by keypusher »

morpheuskafka wrote: post #63

Who is good about not locking accounts or taking similar actions? Apple and Google are both notorious for blocking things for no reason, is AWS or Azure any better?

They are all going to have some form of automated protection against malicious activity, but I suspect AWS and Google’s algorithms are better than the others. My experience with AWS and Google in general is that your treatment varies with your support plan. With business or enterprise level, you have dedicated resources within the company that are going to be aware of such issues or can escalate and sort it out quickly. I understand not wanting to shell out the base cost for enterprise if you are a small company on a budget, but paying at least for business support is a good idea if you are actually running a business. I have never actually heard of this happening to such a customer though, so perhaps they have extra processes in place?

Re: "DigitalOcean Killed Our Company"

#135

Post by znpy »

thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

I have interviewed with DO and they tried diverting me towards a support position.

They told me that on a single day a support engineer was supposed to help/advice customers on pretty much whatever the customer was having issue with and also handle something between 80-120 tickets per day.

It's nice to see that DO is willing to help on pretty much anything they (read: their team) has knowledge about, but with 80-120 tickets per day I cannot expect to give meaningful help.

Needed EDIT: it seems to me that this comments is receiving more attention than it probably deserves, and I feel it's worth clarifying some things:

1. I decided not to move forward with the interview as I was not interested in that support position, so I have not verified that's the volume of tickets.

2. From their description of tickets, such tickets can be anything from "I cannot get apache2 to run" to "how can I get this linucs thing to run Outlook?" (/s) to "my whole company that runs on DO is stuck because you locked my account".

Re: "DigitalOcean Killed Our Company"

#136

Post by cs02rm0 »

thaumaturgy wrote: post #127
_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

That's better than nothing, but still not great. We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data. We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical. Cron jobs can (and do) silently fail in all k…

> We also don't know how big the dataset is or how often it changes.

I'm going to take a stab at small and infrequently.

Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.

Re: "DigitalOcean Killed Our Company"

#138

Post by _bxg1 »

thaumaturgy wrote: post #127
_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

That's better than nothing, but still not great. We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data. We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical. Cron jobs can (and do) silently fail in all k…

I suggested that solution because everyone is saying "they're only a two-man shop so they don't have the time and money to do things properly". Anyone has the time and money to do the above, and there's a 90% chance that it would save them in a situation like this.

Even if they lost some data, even if the backup silently failed and hadn't been running for two months, it's the difference between a large inconvenience and literally your whole business disappearing.

Re: "DigitalOcean Killed Our Company"

#140

Post by PostPost »

coldtea wrote: post #130

> We lost everything, our servers, and more importantly 1 year of database backups. We now have to explain to our clients, Fortune 500 companies why we can’t restore their account. I think what you have to explain is why there wasn't a contingency plan, with your own servers, colocation, another cloud offering, etc...

When AWS has gone down in the past, it's severely impacted massive tech companies like Netflix and Spotify.

Why would there be an expectation that a 2-man shop have "another cloud offering" as a contingency plan when some of the biggest and best tech companies do not?

People use services like AWS or DO because they are the contingency plan - they have the size and scale that smaller companies cannot afford or implement.

Post reply on HN