Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

511–520 of 620 posts

Re: "DigitalOcean Killed Our Company"

#511

Post by xvector »

system2 wrote: post #491
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Should we be concern about our 40+ droplets with DO now? We built our business on DO, we really can go bankrupt as well as our 30+ clients if anything like this happens to us. Please change your support system ASAP otherwise we will be switching to another platform. We are expecting a very serious response from you.

Do not use DO. The very fact that their default automated response to spam is prod downtime is unacceptable.

It requires so many failures in understanding the service being provided across the company for this decision making process to have ever actualized that there is no reasonable expectation of safety or trust from DO at this point.

Re: "DigitalOcean Killed Our Company"

#512

Post by mirimir »

bufferoverflow wrote: post #502
mirimir wrote: post #357

Earlier quoted context omitted.

Will the customer be compensated for business losses?

I hope they sue and win. This bull###t needs to be fought.

IANAL, but DO's ToS is loaded with weasel words.[0] So if they can sue in some jurisdiction where the binding arbitration and liability limitations don't apply, maybe they could at least get a fair settlement.

0) https://www.digitalocean.com/legal/terms-of-service-agreemen...

Re: "DigitalOcean Killed Our Company"

#514

Post by sampo »

lapnitnelav wrote: post #327
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Public post mortem? Brilliant. Hope you can share what you learnt from this incident and hopefully you'll take a hard look at your processes. I'd hate to be caught in the same issue, especially that we are already customers, and I'm not sure I'll have as much clout as Nicolas here to get your attention.

> I'm not sure I'll have as much clout as Nicolas here to get your attention.

Maybe keeping a twitter (and other social media) account with at least a certain number of follower should be considered a part of a company's security strategy? You'd also need to post something interesting periodically, to keep your follower, so that you have their attention when you need it.

Re: "DigitalOcean Killed Our Company"

#515

Post by dfrage »

duxup wrote: post #430
busterarm wrote: post #93

Earlier quoted context omitted.

I accidentally left a 24xlarge instance running for a month without realizing it and they looked at the activity and were totally cool about zeroing the bill for that instance for the month. Basically gave me us a $2000 credit. It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness befor…

The actual cost to Amazon is so low it probably isn't worth insisting on charging the mistakes that contact support.

The good will generated by the stream of customer testimonials of this process we hear about is priceless.

The proposition seems to go something like this: it's a new thing, mistakes are statistically expected, you make an honest one and plead "oops!" and we refund you, no doubt pointing you to resources on best practices and account throttling. As long as the customer takes the lesson to heart, everyone wins.

Re: "DigitalOcean Killed Our Company"

#516

Post by kristopolous »

This is why I always use many providers. I'm talking about $10/month or so packages, this is pretty cheap. They're also not all on the same card in case one gets locked

I use 4 for my current company and have redundancy spread over them so that if, say, AWS goes down or I get an account locked or whatever, nothing is lost, things continue to be operational, slight degradation happens and that's it.

This really isn't that hard to set up. Under a day or so and then just do stuff in ssh config and the shell rc to act as helpers so you remember how to do things.

It's super cheap, pretty easy, and robust.

It's awful what happened to this guy but it's kinda like the person who backs up to the same harddrive as the originals. Awful to lose stuff, it shouldn't happen, but also don't do that.

Re: "DigitalOcean Killed Our Company"

#517

Post by danpalmer »

xiamx wrote: post #428
keypusher wrote: post #179

Given that the author was quite vague about the nature of this “pipeline” and that their product is an “AI-powered Startup Selection engine”, I have a suspicion they were probably crawling and scraping a whole bunch of pages for new startups. It’s possible that this was totally legit and it just looked like a ddos attack, or that it was something else entirely, but everyone here seems to have taken him at his word th…

What's wrong with scraping a bunch of pages. As long as they are following robots.txt, it's no big deal.

Or even better, have contracts with the companies. Maybe unlikely for them, but I think “scraping” is too often assumed to be “bad” in some way. The company I work for does a lot of web scraping, but we have contracts with our partners to scrape their websites. They may still have robots.txt that ask users not to scrape some areas, but we are allowed to bypass those.

Re: "DigitalOcean Killed Our Company"

#519

Post by neop1x »

Maybe they should have their own physical server in a datacenter. In addition to more flexibility, it would also be cheaper. Cloud providers are trying to convince people it is easier by using them, but in the end, if your cloud grows a lot, you are still going to need a team dedicated for management, etc. They are trying to convince it is cheaper to use them, but I can easily have 32GB of RAM or far far more on a single node at the fraction of the cost of the similar virtual offering (if they even offer such big VMs).

Re: "DigitalOcean Killed Our Company"

#520

Post by yardstick »

Are there any best practices I should follow when using AWS, GCP, Azure, DO, and others to avoid these sorts of situations?

I have heard although can’t confirm that using on account billing rather than using a credit card makes them less likely to just disable your account for billing issues. Things like that would be helpful to know. Should I be letting them know more contact info about us, asking for an account manager, etc?

Does it make a difference in how they react to you if you are spending low thousands a month vs tens of thousands a month vs hundreds of thousands etc? Or is everything always automated to death?

Post reply on HN