This same experience happened to me 6mos ago as well, actually I tend to read this online every month or so. No response from them for weeks, they without warning locked and deleted all instances, data, backups, everything. It wasn't until I posted on HN that I got a response. They said it was a mistake and apologized but...yea...whoops we deleted everything and killed your start up, want a free month of service? So no matter how slick the UI, I will go out of my way to never use this trigger happy company that is killing peoples startups on a regular basis.
"DigitalOcean Killed Our Company"
561–570 of 620 posts
Re: "DigitalOcean Killed Our Company"
#562I was evaluating DO the past two months, but reading this, I'm staying at Linode - their poor security incident handling in the past is a theoretical concern, this is a more immediate one.
It's not this anecdote in itself, but that it corroborates my experience during trial that I ignored and dismissed as support incompetence (which should have been a warning sign in itself). After setting up the account, adding a payment card, I wasn't able to enter our VAT ID as part of billing details, with some nondescript error. So I asked support.
Two days(!) later, they responded by asking for incorporation documents, which was frankly bizarre (and a first in ~10 years of running business): they're not exactly a bank with KYC requirements. When I responded, basically, WFT?, and told them to check the billing data in VIES, they eventually fixed it.
But what I got from it was a distinct impression that their default assumption is that the customer is trying to defraud them, even when it makes no sense. To this day, I have no idea what kind of fraud could they possibly be anticipating there (they allowed the card).
This story is on the same general subject, and so are others surfaced here and on twitter in reaction: the customer is presumed scumbag.
Re: "DigitalOcean Killed Our Company"
#563As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…
Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…
With all due respect I think you’ve missed the point. The larger point from my perspective is that you denied your client the ability to move their data off your platform. This would be akin to someone breaking the terms of their lease and you confiscating all their belongings with the intent of burning them. You should provide some sort of grace period for users to move their data off your platform. For everyone else reading this this is should be a wake up call why you should never trust your data to a singular entity. Even if they have 99.9999% uptime you never know when they’ll decide to deny you access to your data.
Re: "DigitalOcean Killed Our Company"
#564Earlier quoted context omitted.
I think it says a lot that this CTO joker flew in, regurgitated the standard-issue "we will endeavor to do better" apology and left without answering any of the very legitimate follow-up questions. I would never deal with an organisation that behaves like these guys.
Is there any response that would satisfy you?
Yes.
"This will not happen again, ever".
People's livelihoods are at stake in DO's hosting. Canned responses and brutal account lockouts should have NEVER been on the table to begin with.
Re: "DigitalOcean Killed Our Company"
#565Earlier quoted context omitted.
Support should be looked at as a profit center, but almost everyone tries to run it like a cost center. It's crazy that companies spend $$ on marketing and sales, then cheap out on a interaction with someone who is already interested in / using their product.
Running profit centers requires comparatively rarer leadership resources while running cost centers only requires easy-to-hire management resources. You don't want your best leaders whipping your support center into shape letting the company's competitive edge fritter away.
Alternatively, I'd ask if you want easy-to-hire management resources as your primary touchpoint with paying customers.
Re: "DigitalOcean Killed Our Company"
#566Re: "DigitalOcean Killed Our Company"
#567Earlier quoted context omitted.
Is there any response that would satisfy you?
Yes. "This will not happen again, ever". People's livelihoods are at stake in DO's hosting. Canned responses and brutal account lockouts should have NEVER been on the table to begin with.
That’d be unrealistic for any company to claim, and if any company I worked with did claim that I would run for the hills.
That’s akin to saying “we’ll never ship a bug”, or “we have an SLO of 100%”. That’s impossible for anyone to claim. Same goes for the response handling. There is clearly a lot of room for improvement there, but if you’re insisting on not getting canned response, that means a human needs to be involved at some point. Humans will at times be slow to respond. Humans will at times make mistakes. This is just an unavoidable reality.
I get that mob mentality is strong when shit hits the fan publicly, but have a bit of empathy and think about what reasonable solutions you may come up with if you were to be in their situation, rather than asking for a “magic bullet”.
I could see a good response here being an overhaul of their incident response policy, especially in terms of L1 support. Probably by beefing up the L2 staffing, and escalating issues more often and more quickly. L2 support is generally product engineers rather than dedicated support staff/contractors, so it’s more expensive to do for sure, but having engineers closer to the “front line” in responding to issues closes the loop better for integrating fixes into the product, and identifying erroneous behavior more quickly.
Re: "DigitalOcean Killed Our Company"
#568This is really scary. Many developers are using Digital Ocean.
Re: "DigitalOcean Killed Our Company"
#569As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…
No offense, as I'm sure this has been hard, but a screwup like this publicly demonstrates DO is not ready for prime time competition against AWS, Azure, GCP and the like.
I'd gladly do whatever it takes to KYC, send you my business license, tax returns, EIN, invoice billing, etc so you know there is someone behind my account.
We spend thousands of hours eliminating single points of failure. If an automated system can undermine that work, DO is not an option for us to host anymore.
Re: "DigitalOcean Killed Our Company"
#570Earlier quoted context omitted.
Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…
What do you recommend your clients to do if that kind of mistake happens to them? Is Twitter-shaming the only way out? I know people say some legal arguments why they close you down and won't say anything, but this is the worst scenario ever. I'd be better off excused at something I didn't do than just ooops we can't tell you anything, your account has been shut down.
This is important. I hate how it had became standard for companies to screw their customers unless they are online-shamed.
The response email even read like a giant polite FUCK YOU (we locked your account, no further action required by you)
You bet I will have further action!
And it is after the shaming that you get an "I am sorry for this situation". Which sounds more like saying "I'm sorry we got caught".
My frustration is not with DO specifically, as they do exactly what every other company does.
But, what of the other thousands of people that got screwed and did not put it on twitter?
It is the equivalent of when you are in a restaurant and get screwed: It is the loudest person that complains more the one that gets the reward, while all the others silently swallow the injustice.