Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

601–610 of 620 posts

Re: "DigitalOcean Killed Our Company"

#601

Post by Phlarp »

zeta0134 wrote: post #595
Phlarp wrote: post #592

Earlier quoted context omitted.

My experiences were all with their "dedicated" or "managed" cloud services. Although I did notice that their marketing seemed to shift in the last months I was working with them for that employer from "let us help you build things on Rackspace" to "let us help you move what you built on Rackspace to AWS"

Yes, the Public Cloud, which houses most of the smaller Managed Infrastructure accounts (minimal support) is one of the bigger ... I believe the polite word is "opportunities?" It's a very pretty UI on top of a somewhat fragile Open Stack deployment, which needs a significant amount of work to patch around noisy infrastructure problems. That turns into a support floor burden, and it shows in ticket latency. Critiques…

So is the business plan now to provide premium support for AWS customers?

Re: "DigitalOcean Killed Our Company"

#602

Post by Lazare »

jeremyjh wrote: post #349
Lazare wrote: post #227

Earlier quoted context omitted.

Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy. > So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3? You're paying to try and ensure you don't need to restore from backups. Our data lives in an RDS cluster (where we pay for read replicas to try and make su…

> Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy. This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script or even something as simple as running a truncate on the wrong table. Having a backup of your database in a different storage medium on the same cloud provid…

> This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script [...] But S3 is already storing your data in three different data centers

That sounds like...the same argument?

A RAID array stores your data on multiple physical drives in the machine, but offers no protection against logical corruption (where you store the same bad data on every drive), destruction of the machine, or loss of access to the machine.

S3 stores your data in multiple physical data centres in the region, but offers no protection against logical corruption, downtime of the entire region, or loss of access to the cloud.

You can't count replicas as providing durability against any threat that will apply equally to all the replicas.

Re: "DigitalOcean Killed Our Company"

#603

Post by marenkay »

bcooks wrote: post #368
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…

As a long term customer here is a small suggestion to make this fail-safe: by default do trust your customers, and just ask them first instead of shooting them down first.

Considering you have been marketing yourself as the platform for developer oriented cloud, you should be aware that surge provisioning can and will always be happening.

Re: "DigitalOcean Killed Our Company"

#604

Post by Moru »

pdimitar wrote: post #579
thanatos_dem wrote: post #578

Earlier quoted context omitted.

I would prefer a generic message and a promise for follow up once all the facts are known over a rushed response that may be incorrect. I’m an engineering manager in an infrastructure team (not at all affiliated with Digital Ocean, tho full disclosure, I do have one droplet for my personal website). I know how postmortems generally work, and it’s messy enough to track down root cause even when it’s not some complex a…

I applaud people like you. Seriously. And I agree with your premise. However, my practice has shown that postmortems are watered-down evasive PR talk, many times. If you look at this through the eyes of a potential startup CTO, wouldn't you be worried about the lack of transparency? And finally, why is such an abrupt account lockdown even on the table, at all? You can't claim you are doing your best when it's very ob…

Our line so far has been to change provider of service if we start getting copy - paste answers from support. We always make sure we can get hold of a human on the phone even without a big uptime contract. This has so far lead us to small companies that are not overrun by free accounts used as spam or SEO accounts. That means they have no need for automatic shutdown of accounts and instead you get a phonecall if something goes wrong.

Re: "DigitalOcean Killed Our Company"

#605

Post by pdimitar »

Moru wrote: post #604
pdimitar wrote: post #579

Earlier quoted context omitted.

I applaud people like you. Seriously. And I agree with your premise. However, my practice has shown that postmortems are watered-down evasive PR talk, many times. If you look at this through the eyes of a potential startup CTO, wouldn't you be worried about the lack of transparency? And finally, why is such an abrupt account lockdown even on the table, at all? You can't claim you are doing your best when it's very ob…

Our line so far has been to change provider of service if we start getting copy - paste answers from support. We always make sure we can get hold of a human on the phone even without a big uptime contract. This has so far lead us to small companies that are not overrun by free accounts used as spam or SEO accounts. That means they have no need for automatic shutdown of accounts and instead you get a phonecall if some…

This is how I would go about it as well. But I imagine that's a big expense for non-small companies, and not only through money but through the time of valuable professionals that could have spend the time improving the bottom line.

I too value less known providers. The human factor in support is priceless.

Re: "DigitalOcean Killed Our Company"

#606

Post by mcny »

EugeneOZ wrote: post #487
dahfizz wrote: post #302

Earlier quoted context omitted.

Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors. And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly…

Lol, only a judge has such rights (to decide if data is illegal or not), not some DO algorithms.

Exactly. Are all images of children illegal? I have a photo of me as an infant. What kind of algorithmic nonsense absolutism are they talking about?

Re: "DigitalOcean Killed Our Company"

#607

Post by nathan-io »

sneak wrote: post #581
dkersten wrote: post #426

Earlier quoted context omitted.

As a DO user who was planning on ramping up usage in the coming weeks and months, this is what scares me and what is making me seriously reconsider.

https://github.com/fog/fog/issues/2525 https://news.ycombinator.com/item?id=6983097 Running anything business or privacy critical on DO is madness.

Indeed, this was bad. I assume they were trying to extend SSD lifetime by reducing writes.

It's fair to note that scrubbing is now the default behavior when a droplet is destroyed, so they did listen to the feedback.

https://ideas.digitalocean.com/ideas/DO-I-1947

Re: "DigitalOcean Killed Our Company"

#608

Post by zeta0134 »

Phlarp wrote: post #601
zeta0134 wrote: post #595

Earlier quoted context omitted.

Yes, the Public Cloud, which houses most of the smaller Managed Infrastructure accounts (minimal support) is one of the bigger ... I believe the polite word is "opportunities?" It's a very pretty UI on top of a somewhat fragile Open Stack deployment, which needs a significant amount of work to patch around noisy infrastructure problems. That turns into a support floor burden, and it shows in ticket latency. Critiques…

So is the business plan now to provide premium support for AWS customers?

Oh, absolutely:

https://www.rackspace.com/managed-aws

This is a big push, internally and externally. I don't know too much about the details (I don't work directly with that team) but it's been one of our bigger talking points for a while now.

Re: "DigitalOcean Killed Our Company"

#609

Post by apple4ever »

marenkay wrote: post #603
bcooks wrote: post #368

Earlier quoted context omitted.

Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns,…

As a long term customer here is a small suggestion to make this fail-safe: by default do trust your customers, and just ask them first instead of shooting them down first. Considering you have been marketing yourself as the platform for developer oriented cloud, you should be aware that surge provisioning can and will always be happening.

This is the right move. Sure, if an account has repeated violations after clear communications, then action must be taken.

But it doesn't make sense to shut it down before discussion!

Re: "DigitalOcean Killed Our Company"

#610

Post by mekane8 »

rdiddly wrote: post #343
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.

So well written. This is exactly what's so scary about this whole thing.
Post reply on HN