"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#421

Post by dahfizz » Sat, Jun 01, 2019, 2:12 AM UTC

shakna wrote: post #404

dahfizz wrote: post #373

Earlier quoted context omitted.

> they should contact law enforcement And do what in the mean time? The legal system acts slowly . In the age of social media outrage, would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO? It's easy to be outraged when these systems and procedures are used against the innocent. That does not mean we should stop using rational thought. If someone is u…

> Your account has been temporarily locked pending the result of an ongoing investigation. You lock down the image, and let law enforcement do their thing. If law enforcement clear them, you then give the customer access to their data, perhaps for a short time before you cut them off as they seem to be a risky customer to have. You don't unilaterally make the decision, you offload your responsibility onto the legal p…

I agree that this was probably the most reasonable decision for them to make.

The fact that there are hundreds of comments on HN condemning them for this action proves my point.

Re: "DigitalOcean Killed Our Company"

#422

Post by macintux » Sat, Jun 01, 2019, 2:13 AM UTC

chris_wot wrote: post #409

dahfizz wrote: post #302

Earlier quoted context omitted.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

At no point did DO ever believe this. This happened purely and simply because of usage patterns changing. It was done automatically and a bot locked them out. They should not be locking out data based on an automated script. You seem to be accusing the aggrieved party of being a bad actor, when that is not the case.

The change in usage patterns does not appear to be the only flag.

https://news.ycombinator.com/item?id=20066331

Re: "DigitalOcean Killed Our Company"

#423

Post by sidlls » Sat, Jun 01, 2019, 2:18 AM UTC

DO didn't handle this we'll, but a company that wants to serve "Fortune 500" customers ought to have a more mature process for handling an outage like this. The fact that they didn't makes it hard to view them as a serious, credible business.

Re: "DigitalOcean Killed Our Company"

#424

Post by jasonjayr » Sat, Jun 01, 2019, 2:22 AM UTC

harshalizee wrote: post #292

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Sure, but the email he received basically said "your account is locked. No other info. Thank You". That to me is a much scarier thing than anything else in the thread. How can anyone trust in your infrastructure if your standard protocol is literally just shutting down their entire operation without any form of review or communication?

We have a relatively large spend ($5k+) @ DO, for a unique client (most of our other clients can be served by our colocated facility), and I'm going to second this. Or with any other provider. They should always explain exactly which rule was broken. If the customer is legit + genuine, they will promptly fix the issue and won't be a further problem. Being vague makes it super troublesome to rely on any service that takes that tactic. (Like Google, for example) If they continue to re-offend, and find other ways to skirt the rules, that's when you move on to account termination.

Re: "DigitalOcean Killed Our Company"

#425

Post by dahfizz » Sat, Jun 01, 2019, 2:27 AM UTC

wwweston wrote: post #402

dahfizz wrote: post #302

Earlier quoted context omitted.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

> If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. The observant will note the particular corner you're backing into here -- that a business might be justified in denying access to code/data being used in literally criminal behavior -- is notably distinct from the general…

> ... a business might be justified in denying access to code/data being used in literally criminal behavior...

I agree. Look at the absolutism of the comment I am replying to. My whole point is that there might be some nuance to the situation.

> ...Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.

Again, I agree. Considering how cheap AWS, backblaze, and Google drive is, it is completely ridiculous to depend on any one single hosting service to hold all your data forever and never err.

Re: "DigitalOcean Killed Our Company"

#426

Post by dkersten » Sat, Jun 01, 2019, 2:30 AM UTC

rdiddly wrote: post #343

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.

As a DO user who was planning on ramping up usage in the coming weeks and months, this is what scares me and what is making me seriously reconsider.

Re: "DigitalOcean Killed Our Company"

#427

Post by robertAngst » Sat, Jun 01, 2019, 2:31 AM UTC

Holy crap. Well that is a story worth sharing.

So... Never trust DigitalOcean for anything important.

Re: "DigitalOcean Killed Our Company"

#428

Post by xiamx » Sat, Jun 01, 2019, 2:32 AM UTC

keypusher wrote: post #179

Given that the author was quite vague about the nature of this “pipeline” and that their product is an “AI-powered Startup Selection engine”, I have a suspicion they were probably crawling and scraping a whole bunch of pages for new startups. It’s possible that this was totally legit and it just looked like a ddos attack, or that it was something else entirely, but everyone here seems to have taken him at his word th…

What's wrong with scraping a bunch of pages. As long as they are following robots.txt, it's no big deal.

Re: "DigitalOcean Killed Our Company"

#429

Post by HeWhoLurksLate » Sat, Jun 01, 2019, 2:34 AM UTC

busterarm wrote: post #73

Hamuko wrote: post #19

Earlier quoted context omitted.

There's a pretty hard cap on the level of redundancy you can do with a two-man company, as I assume a two-man company does not bring in a lot of money.

Their customers are Fortune 500 companies. Clearly they should be charging more.

It might be that if they do that, they could get undercut by someone else.

Re: "DigitalOcean Killed Our Company"

#430

Post by duxup » Sat, Jun 01, 2019, 2:35 AM UTC

busterarm wrote: post #93

kelp wrote: post #62

Earlier quoted context omitted.

This is exactly why AWS has relatively low default account limits, and you have to open a support ticket to raise them. It's largely to prevent run-away costs from surprising the customer.

I accidentally left a 24xlarge instance running for a month without realizing it and they looked at the activity and were totally cool about zeroing the bill for that instance for the month. Basically gave me us a $2000 credit. It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness befor…

The actual cost to Amazon is so low it probably isn't worth insisting on charging the mistakes that contact support.