"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#151

Post by walrus01 » Fri, May 31, 2019, 9:04 PM UTC

nikanj wrote: post #143

thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

Linode is definitely in the minority here. Most companies, in tech and outside of it, seem to follow the DO model. Twitter provides decent service, and the official help channels provide canned responses and template emails. I somewhat blame people in tech, actually. More than one company is creating products that "cut customer service costs via machine learning", which is code for "pick keywords from incoming ticket…

ISP here: The margins in bulk hosting services are incredibly thin, and companies have resorted to automation tools. If somebody asked me to run backend infrastructure for something like DigitalOcean or Linode, I would run away screaming. It would literally be my own personal hell. I would rather run any other sort of ISP services on the planet than a bulkhosting service where anybody with a pulse and $10 to $20/month can sign up for a VPS.

I truly feel sorry for their first and second tier customer support people. I imagine the staff churn rate is incredible.

People who work for these sorts of low-end hosting companies inevitably quit and try to work for an ISP that has more clueful customers. When you have people paying $250/month to colocate a few 1RU servers, the level of clue of the customer and amount of hassle you will get from the customer is a great deal less than a $15/month VPS customer.

Re: "DigitalOcean Killed Our Company"

#152

Post by prophesi » Fri, May 31, 2019, 9:04 PM UTC

PostPost wrote: post #140

coldtea wrote: post #130

> We lost everything, our servers, and more importantly 1 year of database backups. We now have to explain to our clients, Fortune 500 companies why we can’t restore their account. I think what you have to explain is why there wasn't a contingency plan, with your own servers, colocation, another cloud offering, etc...

When AWS has gone down in the past, it's severely impacted massive tech companies like Netflix and Spotify. Why would there be an expectation that a 2-man shop have "another cloud offering" as a contingency plan when some of the biggest and best tech companies do not? People use services like AWS or DO because they are the contingency plan - they have the size and scale that smaller companies cannot afford or impleme…

The difference is that when AWS goes down, Netflix/Spotify still have backups and could adapt infrastructure if the outage involved permanent data-loss. You're talking about the people who built https://github.com/Netflix/chaosmonkey

I'd argue that it should be _easier_ for a 2-man company to adapt to cloud service outages, as they likely don't have to keep up with nearly as many backups or moving parts.

Re: "DigitalOcean Killed Our Company"

#153

Post by nikanj » Fri, May 31, 2019, 9:05 PM UTC

jlv2 wrote: post #40

Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."

On the other hand: Get customers and traction before you build a multi-site, fault-tolerant, self-healing, webscale platform that Google would be proud to have.

I think we needlessly shame one-person operations for focusing on actual customer needs instead of ops busywork and yak shaving.

Re: "DigitalOcean Killed Our Company"

#154

Post by grayed-down » Fri, May 31, 2019, 9:06 PM UTC

We do a fair amount of business with DO to the tune of about 40+ droplets used together and separately for various tasks.

While we could certainly survive the loss of these assets, the recovery would be long and costly.

So I would certainly say that this story gives me a great deal of pause and will take up some mental space this weekend as I think about future dealings with DO.

Re: "DigitalOcean Killed Our Company"

#155

Post by h3throw » Fri, May 31, 2019, 9:06 PM UTC

z3t4 wrote: post #124

_bxg1 wrote: post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

Having backups in two places could easily triple the hosting costs. The question is what costs more. Eg. Losing data vs backup costs.

That seems like an odd cost increase. How do you figure it would lead to a tripling of op costs?

Re: "DigitalOcean Killed Our Company"

#156

Post by yourbandsucks » Fri, May 31, 2019, 9:07 PM UTC

_bxg1 wrote: post #133

stingraycharles wrote: post #122

Earlier quoted context omitted.

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup. Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn th…

And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc.

You can't dot every I, cross every t and also build a compelling product as a 2 person shop.

Re: "DigitalOcean Killed Our Company"

#157

Post by docker_up » Fri, May 31, 2019, 9:07 PM UTC

I thought Fortune 500 companies (at least the enterprise companies I dealt with) that had checklists for their SAAS vendors that had to check off things like Disaster Recovery Readiness, etc.

At the very least this company learned a hard lesson about Disaster Recovery best practices. Hopefully all the up and coming companies reading this story learns as well. Also please remember that a backup that isn't tested IS NOT A BACKUP! I've been in so many situations where backups were corrupted, so part of the disaster recovery is to test the backups and make sure you can really recover.

There are a million different scenarios where their data could be lost and it not be Digital Ocean's fault. It's the company's responsibility to have protected their customers from this.

I worked at a company with no real Disaster Recovery plan. I was told that "we can get the servers up and running within 18 hrs if we had an outage", which not only was absurdly slow but probably an underestimate. Only by the grace of God did we not suffer a real outage but if we did, it was totally the VP's fault for not addressing my concerns.

Re: "DigitalOcean Killed Our Company"

#158

Post by sigstoat » Fri, May 31, 2019, 9:07 PM UTC

CharlesColeman wrote: post #126

badrequest wrote: post #109

Earlier quoted context omitted.

"2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups.

> "2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups. They had backups, but being arbitrarily cut-off from their hosting provider wasn't part of their threat model. Isn't a big part of cloud marketing the idea that they're so good at redundancy, etc. that you don't need to attempt that stuff on your own? The idea that you have to spread your infrastructure across multiple cl…

> In any case, it's also probably too much infrastructure cost for a 2-man company.

keeping your production and your backups in the same cloud provider is the equivalent of keeping your backup tapes right next to the computer they're backing up. you're exposing them both to strongly correlated risks. you've just changed those risks from "fire, water, theft" to "provider, incompetence, security breach"

Re: "DigitalOcean Killed Our Company"

#159

Post by scarface74 » Fri, May 31, 2019, 9:08 PM UTC

busterarm wrote: post #93

kelp wrote: post #62

Earlier quoted context omitted.

This is exactly why AWS has relatively low default account limits, and you have to open a support ticket to raise them. It's largely to prevent run-away costs from surprising the customer.

I accidentally left a 24xlarge instance running for a month without realizing it and they looked at the activity and were totally cool about zeroing the bill for that instance for the month. Basically gave me us a $2000 credit. It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness befor…

There have been so many stories of AWS accounts being “hacked” (actually they weren’t. someone posted their keys to a public github repo), the person panicking, then sending a ticket to AWS and then getting a refund. AWS support is excellent - especially on the business tier and above.

I will gladly pay the extra money for AWS than to even think about DO or even GCP for a money making project.

But more on topic: with Aurora/MySQL you can have an on-site hosted read replica from an AWS hosted database. That would be a cheap, easy real time backup solution if I were really worried about AWS screwing me over.

Re: "DigitalOcean Killed Our Company"

#160

Post by hn_throwaway_99 » Fri, May 31, 2019, 9:08 PM UTC

vbezhenar wrote: post #88

_bxg1 wrote: post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?

Agreed, but it's not like the original poster had a huge platform, he just posted about it on Twitter. I may despise Twitter for a bunch of different reasons, but I can't deny it's a great tool for raising issues to companies.