Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

261–270 of 620 posts

Re: "DigitalOcean Killed Our Company"

#261

Post by bdcravens »

stingraycharles wrote: post #122
_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

> It’s unrealistic to expect this of a tiny startup.

Then it's unrealistic to trust them with your business.

Re: "DigitalOcean Killed Our Company"

#262

Post by tgsovlerkhgsel »

cptskippy wrote: post #241

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

Developer receives an e-mail "we noticed something weird. btw here's our TOS. please explain".

Developer explains as requested.

Developer receives "k, we've restored your account", which sounds an awful lot like "what you're doing is fine".

Developer gets shut down again despite having explained the behavior as requested (i.e. the explanation is on file) and despite the explanation being considered sufficient by DO.

The "with the details you provided, we've removed the hold" e-mail is hard to interpret differently than "sorry for the misunderstanding, this is fine", especially as the initial e-mail asked them to explain to "ensure your account is not subjected to additional scrutiny or placed on billing hold". If they meant "ok but don't do that again" they should have stated so.

Re: "DigitalOcean Killed Our Company"

#263

Post by walrus01 »

yks wrote: post #245
walrus01 wrote: post #173

Earlier quoted context omitted.

Personal opinion, it's really important in the ISP/hosting world to identify what market categories are a race to the bottom, and if at all possible, refuse to participate in them. I look at companies selling $5 to $15/month VPS services and try to figure out how many customers they need to be set up for monthly recurring services, in order to pay for reasonably reliable and redundant infrastructure, and the math jus…

Unfortunately there is no guarantee that I receive any better support at the more expensive provider though

At a certain price point, yes there is, if you're paying $800/month for hosting services to a mid sized regional ISP with presence at major IX points. That ISP cares about its reputation, and cares about the revenue it's getting from you.

I can tell you that as a person whose job title includes "network engineer", we have a number of customers who have critical server/VM functions similar to these people who had the DigitalOcean disaster. If something goes wrong, an actual live human being with at least a moderate degree of linux+neteng clue is going to take a look at their ticket, personally address it, and go through our escalation path if needed.

Re: "DigitalOcean Killed Our Company"

#265

Post by ausjke »

thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

second this, Linode for 15+ years here, tried DO a few times(but never left Linode), now 100% back with Linode.

Linode even has an irc channel(you can use browser to access it), I rarely need support, but when I really need it, it is always fast, to the point, available.

Re: "DigitalOcean Killed Our Company"

#266

Post by lazzlazzlazz »

vbezhenar wrote: post #88
_bxg1 wrote: post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?

To be quite honest and thoughtful about it - probably extremely few companies went bankrupt silently because of issues like this. Let's be realistic.

Re: "DigitalOcean Killed Our Company"

#267

Post by Dylan16807 »

prophesi wrote: post #235
Dylan16807 wrote: post #176

Earlier quoted context omitted.

Okay, then that means they did have a contingency plan, except for a single rsync. Which would mean you disagree strongly with coldtea?

I'm really not sure what your argument is. Their entire business was completely reliant on DO droplets. It doesn't take much foresight to think, "hey, I should probably make a backup in case this VPS goes down." Nothing in this comment thread, or the OP twitter thread mentions anything about the rest of this imaginary contingency plan of theirs.

Coldtea said they need to explain their lack of contingency plan wrt "servers, colocation, another cloud offering, etc...".

PostPost said they didn't, that even huge companies don't have contingency plans.

I agree with PostPost, and I'm trying to figure out which one you agree with.

If you define being able to adapt as a contingency plan, well, I have confidence that this company is fully able to adapt! Their architecture is small and pretty easy to move. The only problem is a lack of external backup, which will be remedied very soon, and once that happens they could easily shift to another service even if DO re-disabled their account.

So that would mean you agree with PostPost. But you don't seem to agree at all.

I'm struggling to reconcile "The ability to adapt is the definition of a contingency plan." and "this imaginary contingency plan of theirs". If you demand a preexisting written plan then that means you're not accepting "the ability to adapt" as a valid answer at all.

Re: "DigitalOcean Killed Our Company"

#268

Post by bdcravens »

SirensOfTitan wrote: post #260
cptskippy wrote: post #241

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

The developer is a 2 person team. Why would they use multiple clouds at that stage? Additionally, if 10 spun up VMs is considered an “unreasonable” load on DigitalOcean infrastructure I shutter at the concept of building anything on the service. Does DigitalOcean even define “unreasonable” in their terms or is it kept vague?

My understanding is that it isn't the 10 VMs as much as the resource usage (my suspicion is that DO is running a lot closer to the margin that larger providers, so they police this more). So they probably pegged all the CPUs at 100%. (perhaps a message queue approach would have been easier on the resources)

Re: "DigitalOcean Killed Our Company"

#269

Post by jturpin »

CharlesColeman wrote: post #126
badrequest wrote: post #109

Earlier quoted context omitted.

"2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups.

> "2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups. They had backups, but being arbitrarily cut-off from their hosting provider wasn't part of their threat model. Isn't a big part of cloud marketing the idea that they're so good at redundancy, etc. that you don't need to attempt that stuff on your own? The idea that you have to spread your infrastructure across multiple cl…

Yeah I think this is what people are not getting. Redundant backups might mean "don't worry, in addition to backups on the instance, I have them going to a S3 bucket in region 1 and then also region 2 in case that region goes down," which of course doesn't protect from malicious activity from the provider. You certainly _should_ make sure you have backups locally available or in a secondary cloud provider but this is some hindsight.

Re: "DigitalOcean Killed Our Company"

#270

Post by bcooks »

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community.

This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happen, and we take every effort to get customers back online as quickly as possible. In this particular scenario, we were slow to respond and had missteps in handling the false positive. This led the user to be locked out for an extended period of time. We apologize for our mistake and will share more details in our public postmortem.

Post reply on HN