Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

341–350 of 620 posts

Re: "DigitalOcean Killed Our Company"

#341

Post by unilynx »

truncate wrote: post #305
cptskippy wrote: post #241

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

`unreasonable load` sounds pretty vague. What counts as unreasonable? 10 VMs doesn't sound like much, and I believe if I'm renting a VM with XYZ specs, I should be allowed to use up-to max capacity it says so in specs. What am I missing here?

Also, digitalocean has a hard droplet (VMs) limit that requires a ticket to raise. You would expect them to be okay with you staying in that limit..

(And you cant just start 10 hugely expensive VMs either, the larger sizes are initially locked too)

Re: "DigitalOcean Killed Our Company"

#342

Post by kbenson »

lapnitnelav wrote: post #327
bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

Public post mortem? Brilliant. Hope you can share what you learnt from this incident and hopefully you'll take a hard look at your processes. I'd hate to be caught in the same issue, especially that we are already customers, and I'm not sure I'll have as much clout as Nicolas here to get your attention.

> and I'm not sure I'll have as much clout as Nicolas here to get your attention.

It's occurring to me now that while I've successfully ignored twitter for years, I should probably rectify that just so I have somewhere to type my hopes and prayers when this eventually happens to me, and hope for a miracle. It sure seems like the only place they're listened to.

Re: "DigitalOcean Killed Our Company"

#343

Post by rdiddly »

bcooks wrote: post #270

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happe…

You've got an additional problem though, which is that this tells us you have two support channels: one that doesn't work (i.e. yours, the one you built), and one that does (Twitter-shaming). The first channel represents how you act when no one's watching; the second, how you act when they are. Most people prefer to deal with people for whom those two are the same.

Re: "DigitalOcean Killed Our Company"

#344

Post by busterarm »

figgis wrote: post #84
busterarm wrote: post #41

Earlier quoted context omitted.

Really? Big European VPS hosts like OVH just turn off your stuff until the problem goes away. Hardly fly-by-night.

Not in my experience they don't. I use OVH and nfoservers and I've had an issue like this exactly once on both hosts. On OVH one of my servers was hacked and running typical scripts that are run once that happens (port checking, common admin credentials, brute force attempts, etc) They cut off all internet access to and from the server and sent me an alert stating what was happening and that I needed to VNC into the…

You said "not in my experience they don't" and then literally describe in detail how they did exactly what I was saying they do.

Re: "DigitalOcean Killed Our Company"

#345

Post by mkhattab »

beejiu wrote: post #102
jlv2 wrote: post #40

Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."

You'll be surprised how many companies are "all in" on AWS or Google Cloud, including ALL backups.

I agree. Who are all these people that pull backups, that maybe GBs or TBs in size for offline storage? How does that even work in practical terms in disaster scenarios like this where resolution times are expected within hours and not days?

Re: "DigitalOcean Killed Our Company"

#346

Post by megaremote »

elcritch wrote: post #324

> After sending multiple emails and DM on Twitter they unlocked our account, we got 12h of downtime and got a nice Wait their entire business was effectively shutdown and all they did was send an email? Granted the DO handling of a possible abuse situation is shocking, but to allow your business to go down for 12 hours and not be trying to call any and every actual human being at DO seems to be negligent on their par…

This is a 2 person startup, one of them was enroute to portugal on holiday.

Re: "DigitalOcean Killed Our Company"

#347

Post by donarb »

bdcravens wrote: post #268
SirensOfTitan wrote: post #260

Earlier quoted context omitted.

The developer is a 2 person team. Why would they use multiple clouds at that stage? Additionally, if 10 spun up VMs is considered an “unreasonable” load on DigitalOcean infrastructure I shutter at the concept of building anything on the service. Does DigitalOcean even define “unreasonable” in their terms or is it kept vague?

My understanding is that it isn't the 10 VMs as much as the resource usage (my suspicion is that DO is running a lot closer to the margin that larger providers, so they police this more). So they probably pegged all the CPUs at 100%. (perhaps a message queue approach would have been easier on the resources)

He did mention Redis, so I assume some sort of message queueing was in play.

Re: "DigitalOcean Killed Our Company"

#348

Post by Aeolun »

AYBABTME wrote: post #219
thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

> DO seems to have gone with the "hire cheap overseas support that almost but doesn't quite understand English" strategy, whereas the tier 1 guys at Linode have on occasion demonstrated more Linux systems administration expertise than I've got. That's simply not true. There's support engineers hired around the world, and depending on when your ticket is posted, someone awake at that time will answer. DO is super remo…

But, isn’t hiring all over the world exactly because it is cheaper for the same kind of talent. I’m sure the company doesn’t do it out of the goodness of their heart.

Then of course, there is no guarantee these people speak and understand english perfectly.

Re: "DigitalOcean Killed Our Company"

#349

Post by jeremyjh »

Lazare wrote: post #227
jeremyjh wrote: post #199

Earlier quoted context omitted.

So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3? I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.

Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy. > So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3? You're paying to try and ensure you don't need to restore from backups. Our data lives in an RDS cluster (where we pay for read replicas to try and make su…

> Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy.

This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script or even something as simple as running a truncate on the wrong table. Having a backup of your database in a different storage medium on the same cloud provider protects from vastly more failure modes.

> Who the hell doesn't have offsite backups?

No one. But S3 is already storing your data in three different data centers even if you have a single bucket in one region, and you also have SQL log replication to another region. Multi-region is as easy as enabling replication but that is only available within a single cloud provider (I can't replicate RDS to Google Cloud SQL, only to another RDS region). I would guess that a lot of people use that rather than using a different cloud provider.

Re: "DigitalOcean Killed Our Company"

#350

Post by DKnoll »

_bxg1 wrote: post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places.

They should have, at the very least, one DR site on a different provider in a different region that is replicated in real-time and ready to go live after an outage is confirmed by the IT Operations team (or automatically depending on what services are being run).

Post reply on HN