"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#571

Post by rat9988 » Sat, Jun 01, 2019, 5:20 PM UTC

nathan-io wrote: post #396

nathan-io wrote: post #295

Earlier quoted context omitted.

Mistakes happen, and algorithms are sometimes a necessary part of scale/efficiency. Everyone understands that. That said, what's highly troubling as a DO customer (and someone who is planning to deploy startup infrastructure of my own with DO) is: 1) The discrepancy between this customer's experience and clear assurances made on this very forum by high-level DO employees that: a. warnings are ALWAYS issued before sus…

I'm no longer able to edit the above comment, so to elaborate on #1: Following is a comment[1] by Moisey Uretsky in another thread[2]: > Depending on which items are flagged the account is put into a locked state, which means that access is limited. However, the droplets for that account and other services are not affected at all. The account is also notified about the action and a dialogue is opened, to determine wh…

What he said in an other thread, and this thread, is press release, marketing. Don't trust what he says to save his business. You have absolutely no reason to.

Re: "DigitalOcean Killed Our Company"

#572

Post by rat9988 » Sat, Jun 01, 2019, 5:22 PM UTC

thanatos_dem wrote: post #474

isoprophlex wrote: post #465

Earlier quoted context omitted.

It's been 7 hours, some follow up answers would be nice...

7 hours, on a Friday night in the headquarters time zone. This issue is resolved and is clearly not wide spread, so does getting a response on Monday or Tuesday vs right now make any difference? Companies are made of people. Let the people have a life. Their night is shitty enough as is after this, I guarantee you.

The thing is, my business don't want to deal with people. It wants to deal with a business made of multiple people to guarantee service availability. If he cannot answer, surely someone else in DigitalOcean can?

Re: "DigitalOcean Killed Our Company"

#573

Post by rat9988 » Sat, Jun 01, 2019, 5:24 PM UTC

amrx431 wrote: post #489

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

A year of Data backup lost. Do you realize how that alone may cause the clients to dump a company and do you realize that startups may never recover from fiasco like these? I understand that it was false positive triggered by internal systems. But how do you explain the delay in restoring the services and reflagging again within hours after the services were restored?

At the end it was not even a matter of delay. It was more like, we locked your account, we don't want to hear you anymore until the end of times.

Re: "DigitalOcean Killed Our Company"

#574

Post by pdimitar » Sat, Jun 01, 2019, 5:56 PM UTC

thanatos_dem wrote: post #567

pdimitar wrote: post #564

Earlier quoted context omitted.

Yes. "This will not happen again, ever". People's livelihoods are at stake in DO's hosting. Canned responses and brutal account lockouts should have NEVER been on the table to begin with.

That’d be unrealistic for any company to claim, and if any company I worked with did claim that I would run for the hills. That’s akin to saying “we’ll never ship a bug”, or “we have an SLO of 100%”. That’s impossible for anyone to claim. Same goes for the response handling. There is clearly a lot of room for improvement there, but if you’re insisting on not getting canned response, that means a human needs to be inv…

Sure, me and a lot of others react rather strongly in these situations. I agree with that but you already seem to understand the reasons.

However, can you say with a straight face that the very generic message left here by DO's CTO instills confidence in you about how will they handle such situations in the future?

Techies hate lawyer/corporate weasel talk. Least that person could do was do their best to speak plainly without promising the sky and the moon.

Re: "DigitalOcean Killed Our Company"

#575

Post by creeble » Sat, Jun 01, 2019, 6:01 PM UTC

Still waiting for response 22 hours in.

The response (and its timing) will determine whether we continue with DO as a host for the (admittedly tiny) bit of infrastructure we host there.

And there had better be a reasonable response on HN if they value their HN-reading customers; it is where we got the first recommendations for them years ago.

Re: "DigitalOcean Killed Our Company"

#576

Post by detaro » Sat, Jun 01, 2019, 6:07 PM UTC

creeble wrote: post #575

Still waiting for response 22 hours in. The response (and its timing) will determine whether we continue with DO as a host for the (admittedly tiny) bit of infrastructure we host there. And there had better be a reasonable response on HN if they value their HN-reading customers; it is where we got the first recommendations for them years ago.

? They have already responded, including directly in this thread.

Re: "DigitalOcean Killed Our Company"

#577

Post by girmad » Sat, Jun 01, 2019, 6:24 PM UTC

Their terms disclaim it, but I'd sue for consequential damages anyway.

IANAL

Re: "DigitalOcean Killed Our Company"

#578

Post by thanatos_dem » Sat, Jun 01, 2019, 6:31 PM UTC

pdimitar wrote: post #574

thanatos_dem wrote: post #567

Earlier quoted context omitted.

That’d be unrealistic for any company to claim, and if any company I worked with did claim that I would run for the hills. That’s akin to saying “we’ll never ship a bug”, or “we have an SLO of 100%”. That’s impossible for anyone to claim. Same goes for the response handling. There is clearly a lot of room for improvement there, but if you’re insisting on not getting canned response, that means a human needs to be inv…

Sure, me and a lot of others react rather strongly in these situations. I agree with that but you already seem to understand the reasons. However, can you say with a straight face that the very generic message left here by DO's CTO instills confidence in you about how will they handle such situations in the future? Techies hate lawyer/corporate weasel talk. Least that person could do was do their best to speak plainl…

I would prefer a generic message and a promise for follow up once all the facts are known over a rushed response that may be incorrect.

I’m an engineering manager in an infrastructure team (not at all affiliated with Digital Ocean, tho full disclosure, I do have one droplet for my personal website). I know how postmortems generally work, and it’s messy enough to track down root cause even when it’s not some complex algorithm like fraud detection going off the rails.

I’d rather get slow information than misinformation, but I understand the frustration in not being able to see the inner working of how an incident is being handled.

Re: "DigitalOcean Killed Our Company"

#579

Post by pdimitar » Sat, Jun 01, 2019, 7:08 PM UTC

thanatos_dem wrote: post #578

pdimitar wrote: post #574

Earlier quoted context omitted.

Sure, me and a lot of others react rather strongly in these situations. I agree with that but you already seem to understand the reasons. However, can you say with a straight face that the very generic message left here by DO's CTO instills confidence in you about how will they handle such situations in the future? Techies hate lawyer/corporate weasel talk. Least that person could do was do their best to speak plainl…

I would prefer a generic message and a promise for follow up once all the facts are known over a rushed response that may be incorrect. I’m an engineering manager in an infrastructure team (not at all affiliated with Digital Ocean, tho full disclosure, I do have one droplet for my personal website). I know how postmortems generally work, and it’s messy enough to track down root cause even when it’s not some complex a…

I applaud people like you. Seriously.

And I agree with your premise. However, my practice has shown that postmortems are watered-down evasive PR talk, many times.

If you look at this through the eyes of a potential startup CTO, wouldn't you be worried about the lack of transparency?

And finally, why is such an abrupt account lockdown even on the table, at all? You can't claim you are doing your best when it's very obvious that you are just leaving your customers at the mercy of very crude algorithms -- and those, let's be clear on that, could have been created without ever locking down an account without a human approval at the final step.

What I'm saying is that even at this early stage when we know almost nothing, it's evident that this CTO here is not being sincere. It seems DO just wants to maximally automate away support, to their customers' detriment.

Whatever the postmortem ends up being it still won't change the above.

Re: "DigitalOcean Killed Our Company"

#580

Post by creeble » Sat, Jun 01, 2019, 7:15 PM UTC

detaro wrote: post #576

creeble wrote: post #575

Still waiting for response 22 hours in. The response (and its timing) will determine whether we continue with DO as a host for the (admittedly tiny) bit of infrastructure we host there. And there had better be a reasonable response on HN if they value their HN-reading customers; it is where we got the first recommendations for them years ago.

? They have already responded, including directly in this thread .

>We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short.

I'm referring to _this_ response, i.e., the one where they explain why they did what they did.