"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#481

Post by EugeneOZ » Sat, Jun 01, 2019, 7:39 AM UTC

v4ult wrote: post #230

rapind wrote: post #196

Earlier quoted context omitted.

I've been on Linode for 8+ years now (moved there from Slicehost when Rackspace swallowed them up) and their service (not necessarily customer support) has significantly degraded. Not sure I blame them though. They've become far more popular since I started with them and are probably doing their best to grow... but I no longer recommend them as I used to. Just my experience though.

So who would you recommend?

If you are looking for high availability - Google Cloud Platform network is the best. In some other things AWS is better, but GCP network quality is awesome.

Re: "DigitalOcean Killed Our Company"

#482

Post by Cakez0r » Sat, Jun 01, 2019, 7:44 AM UTC

chris_wot wrote: post #401

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

It's not the false positive that is the issue here. The issue is that a. it took way too long to get the business back up and running, and b. the second response gave no explanation and no recourse for the business to become operational again. The very fact that this can happen from an automated script with no oversight should give every one of your customers pause as to whether they continue with your service.

I'd say the issue is that DO is shutting down servers for any reason at all (legal issues aside). If DO sells a product with a particular capacity, why should they intervene at all if a user is using all of the capacity they're paying for?

Re: "DigitalOcean Killed Our Company"

#483

Post by MagicPropmaker » Sat, Jun 01, 2019, 7:44 AM UTC

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

So unless a person is popular enough to get enough people talking about it on twitter or hacker news, someone whose account is flagged by your bad script is going to lose his business.

That doesn't sound good to me.

Re: "DigitalOcean Killed Our Company"

#484

Post by davidgerard » Sat, Jun 01, 2019, 7:45 AM UTC

Blackthorn wrote: post #442

walrus01 wrote: post #173

Earlier quoted context omitted.

Personal opinion, it's really important in the ISP/hosting world to identify what market categories are a race to the bottom, and if at all possible, refuse to participate in them. I look at companies selling $5 to $15/month VPS services and try to figure out how many customers they need to be set up for monthly recurring services, in order to pay for reasonably reliable and redundant infrastructure, and the math jus…

Ten years ago, I had a pretty reasonable $10/mo account, that I eventually moved to a $20/mo account because I needed more resources to keep up with traffic. I'd expect that, as more transistors have been packed blah blah blah, that such a $10/mo account would have gotten better , not worse, since then.

> I'd expect that, as more transistors have been packed blah blah blah, that such a $10/mo account would have gotten better, not worse, since then.

This is what Linode do - they keep you on the same payment level, but raise what you get for it.

Re: "DigitalOcean Killed Our Company"

#485

Post by chris_wot » Sat, Jun 01, 2019, 7:47 AM UTC

macintux wrote: post #422

chris_wot wrote: post #409

Earlier quoted context omitted.

At no point did DO ever believe this. This happened purely and simply because of usage patterns changing. It was done automatically and a bot locked them out. They should not be locking out data based on an automated script. You seem to be accusing the aggrieved party of being a bad actor, when that is not the case.

The change in usage patterns does not appear to be the only flag. https://news.ycombinator.com/item?id=20066331

No, it was one of a number of factors. Usage pattern was definitely a factor. It’s still pretty awful.

Re: "DigitalOcean Killed Our Company"

#486

Post by nurettin » Sat, Jun 01, 2019, 7:52 AM UTC

znpy wrote: post #211

nurettin wrote: post #171

Earlier quoted context omitted.

7 tireless hours of work (with lunch break) 15 minutes to Listen, understand and resolve an issue, assuming perfect knowledge, a lot of luck and normal human speed, that would still amount to less than 30 resolutions a day.

something's not clear: are you talking from direct experience ?

I have no prior experience working at DO tech support.

Re: "DigitalOcean Killed Our Company"

#487

Post by EugeneOZ » Sat, Jun 01, 2019, 7:52 AM UTC

dahfizz wrote: post #302

jwr wrote: post #286

Earlier quoted context omitted.

Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

Lol, only a judge has such rights (to decide if data is illegal or not), not some DO algorithms.

Re: "DigitalOcean Killed Our Company"

#488

Post by zhte415 » Sat, Jun 01, 2019, 7:53 AM UTC

michaelmrose wrote: post #469

zhte415 wrote: post #461

Earlier quoted context omitted.

I don't know the data you're holding. If it is sensitive data, like customer anything, would it infact make sense not to have offsite backup? Reasoning: Your contract with Amazon promises durability and I'm sure there's a service level agreement with penalty/liability clauses. By implementing a redundant backup, you're replicating something that you don't legally need to have, double-or-more due diligence on the offs…

Regardless of durability if you lose your customers data are you sure you will have customers paying you to keep you in business while you figure out liability?

In this case, it was not losing data, but losing access to data. The data was eventually restored. Lose customers' data could also mean losing the backup:

"We're sorry, the tape that we didn't needed to keep has been lost/zero-dayed/secondary service provider has gone bankrupt/Billy's house that we left it at got robbed." These must be disclosed to a customer immediately.

Minimising attack/liability surface is not only a technical problem, but a business one too.

Re: "DigitalOcean Killed Our Company"

#489

Post by amrx431 » Sat, Jun 01, 2019, 7:58 AM UTC

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

A year of Data backup lost. Do you realize how that alone may cause the clients to dump a company and do you realize that startups may never recover from fiasco like these? I understand that it was false positive triggered by internal systems. But how do you explain the delay in restoring the services and reflagging again within hours after the services were restored?

Re: "DigitalOcean Killed Our Company"

#490

Post by janjanson » Sat, Jun 01, 2019, 8:00 AM UTC

This is probably going to get buried in the replies, but I had a similar experience with DigitalOcean about a year ago with my account getting permanently locked with very little explanation and no way of getting it back. It's still locked to this day. I was just a student using my Github student package credit, but I was pretty appalled by the service from DO and vowed never to buy from them.

Unfortunately, my ticket no longer appears under closed tickets. I was still able to dig up my original ticket message and all the responses their support made to me through my email though. Here they are:

https://imgur.com/a/NJMRnyY

Between the replies I asked about what I could do to verify my account. As you can see, they didn't even give me a single chance to do so. They told me to hang on twice then just permanently closed it up. I'm not sure how I even got flagged. All I did was turn on a droplet and delete it. I checked the audit logs, and there was nothing suspicious there either. It was just me logging in and out.

I thought about making a deal about it on Twitter, but I didn't bother because I don't have any followers and it wasn't a huge loss to me either. Maybe it's the only way?