Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

141–150 of 620 posts

Re: "DigitalOcean Killed Our Company"

#141
I see a lot of people saying "They should have had more than one provider" and "They should have had better backups".

What I'm saying is: If you ARE going to put your business at the mercy of one company from top to bottom, would it not be wise to try to get some kind of account rep? Or have some sort of communication with the company as to the nature of your operation?

And if that isn't an option, should you do business with them?

Re: "DigitalOcean Killed Our Company"

#142
post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.

Re: "DigitalOcean Killed Our Company"

#143

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

Linode is definitely in the minority here. Most companies, in tech and outside of it, seem to follow the DO model. Twitter provides decent service, and the official help channels provide canned responses and template emails.

I somewhat blame people in tech, actually. More than one company is creating products that "cut customer service costs via machine learning", which is code for "pick keywords from incoming tickets and autoreply with a template"

Re: "DigitalOcean Killed Our Company"

#144
post #106

If you're reading this and concerned for your own backup story, fret not! In 2019 secure off-premise backups are super easy to implement, even for a 1 person shop. Get something like Restic or Borg or any one of the enumerated options here: https://github.com/restic/others I've recently implemented backups with Restic, the static binary and plethora of supported storage backends was extremely appealing. The easiest s…

> The easiest seems to be to just point it at a S3 bucket, but given most people have infrastructure on AWS (off-premise means off-premise) having other options supported out of the box is pretty handy. Sure, having other options is good: https://min.io/

Minio is great! I use it! But spinning up and managing yet another service when you're already a small shop adds more barriers to entry. Maybe find an already s3-compatible store (like Wasabi) or find something cheap and easy to spin up that's supported by the tool (like https://www.hetzner.com/storage/storage-box)

Re: "DigitalOcean Killed Our Company"

#145
post #40

Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."

This is conflating two different things. One point is valid, the other is not.

- No offsite backups? Agreed. Even for a two person team it is sloppy.

- "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinking with no basis in reality.

Plus if they did over-engineer it by making it cloud agnostic, setting up accounts to sit dormant, cold instances elsewhere, etc, people would just criticize them for that inefficiency/wasting time.

Re: "DigitalOcean Killed Our Company"

#146
post #143

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

Linode is definitely in the minority here. Most companies, in tech and outside of it, seem to follow the DO model. Twitter provides decent service, and the official help channels provide canned responses and template emails. I somewhat blame people in tech, actually. More than one company is creating products that "cut customer service costs via machine learning", which is code for "pick keywords from incoming ticket…

There’s nothing wrong with that approach, the person raising the support ticket likely hasn’t read through all the documentation of the product they’re using.

Re: "DigitalOcean Killed Our Company"

#147

Earlier quoted context omitted.

That's better than nothing, but still not great. We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data. We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical. Cron jobs can (and do) silently fail in all k…

> We also don't know how big the dataset is or how often it changes. I'm going to take a stab at small and infrequently. Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.

Yeah, probably. But we shouldn't be calling these guys out for not taking the "obvious and simple" solution when we aren't 100% certain that it would actually work. That happens too often on HN, and then sometimes the people involved pop in to explain why it's not so simple, and everyone goes "...oh." Seems like we should learn something from that. I've gone with "don't assume it's as simple as your ego would lead you to believe."

Re: "DigitalOcean Killed Our Company"

#148
post #6

I’m certainly getting downvoted for this. Not at all trying to blame the original posters and victims: While they seem great for hobbyist and small business sites, there’s no way I’d trust Fortune 500 client business to something like DigitalOcean. I just don’t see the benefits over a more established operation like AWS, Azure or GCP. Saving $50 here and there isn’t worth it.

Your "established operations" are more expensive and GCP is newer than DO. Plus, getting the attention of someone to restore your account is probably easier in DO than in a faceless giant company like Amazon, Microsoft or Google.

You Get What You Pay For.

Yes, this is a bad look for DO. But the way they're able to beat AWS on price includes things like "worse support." And if AWS goes out of business, you'll know in advance. DO isn't the same story. You should be planning for redundancy if DO is truly business critical for you.

Re: "DigitalOcean Killed Our Company"

#149
post #135

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

I have interviewed with DO and they tried diverting me towards a support position. They told me that on a single day a support engineer was supposed to help/advice customers on pretty much whatever the customer was having issue with and also handle something between 80-120 tickets per day. It's nice to see that DO is willing to help on pretty much anything they (read: their team) has knowledge about, but with 80-120…

Yikes. Those metrics are entirely unreasonable for providing any support besides canned responses. Rough seas at DO.

Disclaimer: Based on providing support myself and coaching a support team at both a web hosting company and ISP I used to own years ago.

Re: "DigitalOcean Killed Our Company"

#150
post #94
post #6

I’m certainly getting downvoted for this. Not at all trying to blame the original posters and victims: While they seem great for hobbyist and small business sites, there’s no way I’d trust Fortune 500 client business to something like DigitalOcean. I just don’t see the benefits over a more established operation like AWS, Azure or GCP. Saving $50 here and there isn’t worth it.

I think it's the opposite. The cofounder intervened and the access was restored. This would never happen with Google or Amazon -- once they lock you out, your entire business is permabanned, and you won't be able to reach any human with authority to help you.

Maybe - AWS at least, you'll have an account rep to bang on whose job it is to remove obstacles to your spending more. I'm not sure how big exactly you have to be to get an account rep, but my small company with only 50 instances has one.
Post reply on HN