"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#401

Post by chris_wot » Sat, Jun 01, 2019, 1:56 AM UTC

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

It's not the false positive that is the issue here. The issue is that a. it took way too long to get the business back up and running, and b. the second response gave no explanation and no recourse for the business to become operational again.

The very fact that this can happen from an automated script with no oversight should give every one of your customers pause as to whether they continue with your service.

Re: "DigitalOcean Killed Our Company"

#402

Post by wwweston » Sat, Jun 01, 2019, 1:57 AM UTC

dahfizz wrote: post #302

jwr wrote: post #286

Earlier quoted context omitted.

Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

> If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it.

The observant will note the particular corner you're backing into here -- that a business might be justified in denying access to code/data being used in literally criminal behavior -- is notably distinct from the general and likely much more common case.

> they can do whatever they want with their own hard drives.

Sure. But to the extent they take that approach, Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.

Businesses that can be relied on generally instead offer terms of service and processes that don't really allow them to act arbitrarily.

Re: "DigitalOcean Killed Our Company"

#403

Post by [deleted] » Sat, Jun 01, 2019, 1:59 AM UTC

rdiddly wrote: post #343

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.

[deleted]

Re: "DigitalOcean Killed Our Company"

#404

Post by shakna » Sat, Jun 01, 2019, 2:00 AM UTC

dahfizz wrote: post #373

p2t2p wrote: post #325

Earlier quoted context omitted.

When this happens they should contact law enforcement, not play god.

> they should contact law enforcement And do what in the mean time? The legal system acts slowly . In the age of social media outrage, would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO? It's easy to be outraged when these systems and procedures are used against the innocent. That does not mean we should stop using rational thought. If someone is u…

> Your account has been temporarily locked pending the result of an ongoing investigation.

You lock down the image, and let law enforcement do their thing. If law enforcement clear them, you then give the customer access to their data, perhaps for a short time before you cut them off as they seem to be a risky customer to have.

You don't unilaterally make the decision, you offload your responsibility onto the legal process.

Re: "DigitalOcean Killed Our Company"

#405

Post by rapind » Sat, Jun 01, 2019, 2:01 AM UTC

v4ult wrote: post #230

rapind wrote: post #196

Earlier quoted context omitted.

I've been on Linode for 8+ years now (moved there from Slicehost when Rackspace swallowed them up) and their service (not necessarily customer support) has significantly degraded. Not sure I blame them though. They've become far more popular since I started with them and are probably doing their best to grow... but I no longer recommend them as I used to. Just my experience though.

So who would you recommend?

I don't really have a low budget alternative. I know that for our service we're evaluating both google and amazon cloud offerings, but only for our high availability services. I figure DO is in the same boat if not worse.

Re: "DigitalOcean Killed Our Company"

#406

Post by ryanackley » Sat, Jun 01, 2019, 2:02 AM UTC

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

I'm genuinely curious. What type of fraud or abuse are you trying to prevent? Maybe cover that in the postmortem.

Re: "DigitalOcean Killed Our Company"

#407

Post by jjeaff » Sat, Jun 01, 2019, 2:03 AM UTC

AcerbicZero wrote: post #215

Someone1234 wrote: post #145

Earlier quoted context omitted.

This is conflating two different things. One point is valid, the other is not. - No offsite backups? Agreed. Even for a two person team it is sloppy. - "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinkin…

Some may have an availability dependency on those services, but if they don't have a full BC and DR plan ready to go within a few hours of losing those service they're not going to be a big enterprise for long.

Are you talking about a small WordPress site or something?

Very, very few tech companies could simply move everything to a new cloud provider in a few hours. I would even hazard a guess that almost none can.

I have all my infrastructure as code and can break it all down and spin it back up in kubernetes clusters in minutes. But due to the quirks of each cloud provider, there are tons of little fixes that would inevitably need to be made.

Not to mention that many companies have way more data than could even be copied over in a few hours.

Re: "DigitalOcean Killed Our Company"

#408

Post by lioeters » Sat, Jun 01, 2019, 2:03 AM UTC

bcooks wrote: post #368

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…

Thank you for jumping in personally to clarify what happened.

As a business owner with much of our infrastructure depending on DigitalOcean, the incident is concerning. It affects the reputation of DO as well as its customers.

The demographics on Twitter and especially here on HN represents a sizable crowd with decision-making influence on DO's bottom line. I hope to see some effort being made to prevent situations like this in the future, and to regain the trust.

As a (so far) satisfied customer, it's great to hear that:

> A combination of factors, not just the usage patterns, led to the initial flag.

> We recognize and embrace our customers ability to spin up highly variable workloads, which would normally not lead to any issues.

> we are looking into our process and how we responded so we can improve upon this

Re: "DigitalOcean Killed Our Company"

#409

Post by chris_wot » Sat, Jun 01, 2019, 2:03 AM UTC

dahfizz wrote: post #302

jwr wrote: post #286

Earlier quoted context omitted.

Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

At no point did DO ever believe this. This happened purely and simply because of usage patterns changing. It was done automatically and a bot locked them out. They should not be locking out data based on an automated script.

You seem to be accusing the aggrieved party of being a bad actor, when that is not the case.

Re: "DigitalOcean Killed Our Company"

#410

Post by nemothekid » Sat, Jun 01, 2019, 2:03 AM UTC

dahfizz wrote: post #373

p2t2p wrote: post #325

Earlier quoted context omitted.

When this happens they should contact law enforcement, not play god.

> they should contact law enforcement And do what in the mean time? The legal system acts slowly . In the age of social media outrage, would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO? It's easy to be outraged when these systems and procedures are used against the innocent. That does not mean we should stop using rational thought. If someone is u…

>would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO?

Seems to work just fine for AWS, Google and Cloudflare. In fact, counter to your argument, Cloudflare got in massive shit when they did decide to play God.