Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

251–260 of 620 posts

Re: "DigitalOcean Killed Our Company"

#251

Earlier quoted context omitted.

7 tireless hours of work (with lunch break) 15 minutes to Listen, understand and resolve an issue, assuming perfect knowledge, a lot of luck and normal human speed, that would still amount to less than 30 resolutions a day.

with no data or experience to back this up, i'd expect this to be kind of a long tail, where some will take an hour, and the majority will take 30 sec

having data and experience to back this up --

you are entirely correct.

Re: "DigitalOcean Killed Our Company"

#252

Earlier quoted context omitted.

7 tireless hours of work (with lunch break) 15 minutes to Listen, understand and resolve an issue, assuming perfect knowledge, a lot of luck and normal human speed, that would still amount to less than 30 resolutions a day.

with no data or experience to back this up, i'd expect this to be kind of a long tail, where some will take an hour, and the majority will take 30 sec

Yep, this is spot on - I used to work on a webhosting help desk and could bang out about 100 tickets a shift, because so many were small queries that required no depth work.

Re: "DigitalOcean Killed Our Company"

#253
I agree with the other comments about not having backups in the same place, and ensuring that you distribute your assets (domain, DNS, compute, backups, etc.) across as many providers as possible.

One thing I will add is that, especially for a small shop or project, assume from the get go that by renting infrastructure from DO (or any provider) that user-hostile actions can and will be taken when it comes to any issue regarding TOS violations that you are unaware of.

This assumption helps to build redundancy in your mindset. Have a production website or app in DO for example? Droplet backup, periodic snapshots, B2 server backup, S3 tar backups, containerize apps if possible, have equally provisioned (smaller, idle VMs) infra on another DO account or another provider if possible, and so on. I know this is overkill but paranoid sysadmins/devops are always rewarded.

Just to add some context for DO specifically, they're a great provider in my anecdotal experience and they are constantly rolling out services aimed at medium to large scale workloads, such as managed databases and k8s.

That being said, it's entirely possible to transfer snapshots [0] to another DigitalOcean user account or teams account. So at the very least, create an entirely new DO account just for holding snapshots, outside of the native droplet backups and third-party backups you're doing on an application level.

[0] https://www.digitalocean.com/docs/images/snapshots/how-to/ch...

Re: "DigitalOcean Killed Our Company"

#254
post #64

Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.

Uretsky:

> Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time.

So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."

Re: "DigitalOcean Killed Our Company"

#255

Given that the author was quite vague about the nature of this “pipeline” and that their product is an “AI-powered Startup Selection engine”, I have a suspicion they were probably crawling and scraping a whole bunch of pages for new startups. It’s possible that this was totally legit and it just looked like a ddos attack, or that it was something else entirely, but everyone here seems to have taken him at his word th…

Even if there is legitimate cause to disable the account, I would prefer to see DO being more transparent with their process here.

Re: "DigitalOcean Killed Our Company"

#256
post #243

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?

I guess it depends on what each of those VMs is doing. I don't trust the Dev's explanation tbh.

> to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.

Tildes preceding numbers means "approximately". Why doesn't he know exactly how many VMs he spun up?

Was he actually spinning up 1 VM per record and only allowing 10 VMs to be running concurrently?

I'm not a Pythong dev but why can't you execute 10 instances of Python on a single VM?

If you need to dedicate an entire VM to processing 1 row, what the hell is it doing?

Re: "DigitalOcean Killed Our Company"

#257
post #84

Earlier quoted context omitted.

Really? Big European VPS hosts like OVH just turn off your stuff until the problem goes away. Hardly fly-by-night.

Not in my experience they don't. I use OVH and nfoservers and I've had an issue like this exactly once on both hosts. On OVH one of my servers was hacked and running typical scripts that are run once that happens (port checking, common admin credentials, brute force attempts, etc) They cut off all internet access to and from the server and sent me an alert stating what was happening and that I needed to VNC into the…

I run the IPFS daemon on Hetzner, and it was trying to connect to local IPs because of some misconfiguration. They sent me an email saying my server was portscanning their LAN, and I should fix it and email them how I fixed it.

I didn't know what they were talking about so I replied saying that, they helped me shut down the local port connections and I never heard another complaint from them. There was no downtime or banning at any point.

Re: "DigitalOcean Killed Our Company"

#258
As someone who has spent the last 8 years in big enterprise it amuses me slightly that small businesses are discovering this now. Because where I work our customers often have contracts that make us accountable for every single hour of downtime. Requiring us to pay customers penalty fees if their service level is affected.

Re: "DigitalOcean Killed Our Company"

#259
post #243

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?

Smaller VPS providers like Linode or DO oversubscribe like crazy. Last time I used Linode, they would email us telling us we're using too much CPU or memory, and we'd need to move to a larger tier VM.

Re: "DigitalOcean Killed Our Company"

#260

Let me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "H…

The developer is a 2 person team. Why would they use multiple clouds at that stage?

Additionally, if 10 spun up VMs is considered an “unreasonable” load on DigitalOcean infrastructure I shutter at the concept of building anything on the service. Does DigitalOcean even define “unreasonable” in their terms or is it kept vague?

Post reply on HN