Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

611–620 of 620 posts

Re: "DigitalOcean Killed Our Company"

#611

Post by apple4ever »

janjanson wrote: post #490

This is probably going to get buried in the replies, but I had a similar experience with DigitalOcean about a year ago with my account getting permanently locked with very little explanation and no way of getting it back. It's still locked to this day. I was just a student using my Github student package credit, but I was pretty appalled by the service from DO and vowed never to buy from them. Unfortunately, my ticke…

Wow that is just terrible.

"We've tried nothing and we are all out of ideas! LOCKED" is basically their response.

I'd complain on Twitter just to see what happens.

Re: "DigitalOcean Killed Our Company"

#612

Post by bcooks »

bcooks wrote: post #368
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…

I wanted to provide you all with an update on the postmortem I promised on Friday. Our analysis has been completed. We will be sharing the full document soon and will publish a link in this thread for those wanting to read it. We promised Raisup a first look and we have provided the draft document to them this afternoon. Because some information in the document could be considered sensitive we wanted to give Raisup a chance to review the document before sharing with the public.

Re: "DigitalOcean Killed Our Company"

#613

Post by sneak »

nathan-io wrote: post #607
sneak wrote: post #581

Earlier quoted context omitted.

https://github.com/fog/fog/issues/2525 https://news.ycombinator.com/item?id=6983097 Running anything business or privacy critical on DO is madness.

Indeed, this was bad. I assume they were trying to extend SSD lifetime by reducing writes. It's fair to note that scrubbing is now the default behavior when a droplet is destroyed, so they did listen to the feedback. https://ideas.digitalocean.com/ideas/DO-I-1947

The SSD thing is a red herring.

You do not need to scrub or write anything to not provide user A’s data to user B in a multi-tenant environment. Sparse allocation can easily return nulls to a reader even while the underlying block storage still contains the old data.

They were just incompetent.

On top of all of that, when I pointed out that what they were doing was absolute amateur hour clownshoes, they oscillated between telling me it was a design decision working as intended (and that it was fine for me to publicize it), and that I was an irresponsible discloser by sharing a vulnerability.

Then they made a blog post lying about how they hadn’t leaked data when they had.

Nope.

Re: "DigitalOcean Killed Our Company"

#614

Post by chaostheory »

yeukhon wrote: post #378
chaostheory wrote: post #360

Earlier quoted context omitted.

Does DO have different levels of support that you can pay for like AWS? I like that system. You pay when you need it. You pay more if you need more support. The difference in support DO vs Linode is probably due to DO being cheaper.

AWS support is fixed contract. There is basic, business and enterprise. What do you mean by “you pay when you need it”?

I can cancel the fixed contract any time. e.g. when I need support I buy a month of basic or business.

Re: "DigitalOcean Killed Our Company"

#615

Post by bcooks »

LeonM wrote: post #499
ryanackley wrote: post #406

Earlier quoted context omitted.

I'm genuinely curious. What type of fraud or abuse are you trying to prevent? Maybe cover that in the postmortem.

If your DO (or other cloud provider) credentials are compromised, it's usually a matter of seconds before someone fires up the largest possible number of instances to start crypto mining.

Yup. LeonM, you are correct. In this case that was the cryptocurrency mining detector that was triggered. More details in the postmortem.

Re: "DigitalOcean Killed Our Company"

#616

Post by bcooks »

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Last week ended on a real low note for many of us at DO. We took a perfectly good customer and gave them an experience no one should have to go through (all while he was trying to leave on vacation no less). We can and must do better. To do better we need to learn from our mistakes. To that end, we also think sharing the information about this incident openly is the best way to help all our customers understand what happened and what we are doing to prevent it in the future.

Yesterday we completed our postmortem analysis of the incident involving Nicolas (@w3Nicolas) and his company Raisup (@raisupcom). With their permission we are sharing the full report on our blog here:

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

Re: "DigitalOcean Killed Our Company"

#618

Post by bogomipz »

Twirrim wrote: post #462
bogomipz wrote: post #398

Earlier quoted context omitted.

Agreed. I actually had to reread that a few times because I could not believe that someone actually approved of that text. That text suggests larger organizational problems within the company.

If you explain why, you give the actual abusers clues as to how to avoid detection. It's common place behaviour for companies to _not_ reveal details.

You can provide a helpful message with options for recourse without giving abuser's "clues." These are not somehow mutually exclusive. By your logic it makes sense to punish a marginal element at the expense of the majority.

Re: "DigitalOcean Killed Our Company"

#619

Post by bofadeez »

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Unacceptable. I just instructed my team to begin a transition to AWS.

Re: "DigitalOcean Killed Our Company"

#620

Post by bofadeez »

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

You ruined your brand
Post reply on HN