[deleted]
"DigitalOcean Killed Our Company"
271–280 of 620 posts
Re: "DigitalOcean Killed Our Company"
#272Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
Uretsky: > Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time. So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."
How would you like that to have been addressed?
I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.
Re: "DigitalOcean Killed Our Company"
#273Earlier quoted context omitted.
with no data or experience to back this up, i'd expect this to be kind of a long tail, where some will take an hour, and the majority will take 30 sec
Yep, this is spot on - I used to work on a webhosting help desk and could bang out about 100 tickets a shift, because so many were small queries that required no depth work.
Old MSFT rule of thumbs was 2 bugs per day during bug crunch mode. Sounds crazy, but when you consider the number of "this text is wrong" and "that text box is too short" bugs that existed after a year of furious development, it wasn't too hard to achieve.
Gotta hit that ZBB!
Re: "DigitalOcean Killed Our Company"
#274Earlier quoted context omitted.
>getting locked out of your account is usually just not a consideration How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan. Dumping everything to B2 is trivially easy, triv…
> How many horror stories need to reach the front page of HN before people stop believing this Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
Even if the stories we hear of account lockouts isn't typical, the absolute number of them that we see -- especially those (like this one) that appear to be locked (and re-locked) by automated processes -- should be cause for concern when setting up a new business on someone else's infrastructure.
Re: "DigitalOcean Killed Our Company"
#275Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
Having backups in two places could easily triple the hosting costs. The question is what costs more. Eg. Losing data vs backup costs.
Most of the costs aren't from storage space, but compute power. We aren't talking about duplicating the whole infrastructure, just backing up the data. Disk space is dirt cheap.
Also, by "two places" I meant the live DB and one backup that's somewhere completely different. My wording may have been confusing.
Re: "DigitalOcean Killed Our Company"
#276I would love to hear what would all the advocates of going "100% in the cloud" think about this.
Re: "DigitalOcean Killed Our Company"
#277Earlier quoted context omitted.
Lets say you have 100 TB of data, plus two backups, you are now paying for 300 TB of data.
I would say that it doubles the cost of backups, but using this math, we start with one copy plus one backup, and add a second backup; that means only a 50% increase.
Also a secondary backup doesn't need to be in hot storage. Coldline or Glacier or similar can easily be a quarter of the price per GB.
Re: "DigitalOcean Killed Our Company"
#278Earlier quoted context omitted.
Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?
Smaller VPS providers like Linode or DO oversubscribe like crazy. Last time I used Linode, they would email us telling us we're using too much CPU or memory, and we'd need to move to a larger tier VM.
I think you misunderstood those emails. They are just there to help you if you didn't realize some process was stuck or something, they specifically say "This is not meant as a warning or a representation that you are misusing your resources." and you can also change the value that triggers those emails or disable them completely.
Re: "DigitalOcean Killed Our Company"
#279Looks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.
That's probably a reason to use containerization / other technologies so that you can spin up your services in a couple minutes on a different cloud provider.
Re: "DigitalOcean Killed Our Company"
#280Earlier quoted context omitted.
Smaller VPS providers like Linode or DO oversubscribe like crazy. Last time I used Linode, they would email us telling us we're using too much CPU or memory, and we'd need to move to a larger tier VM.
I think you misunderstood those emails. They are just there to help you if you didn't realize some process was stuck or something, they specifically say "This is not meant as a warning or a representation that you are misusing your resources." and you can also change the value that triggers those emails or disable them completely.
No it was definitely a ticket from their support, telling us we were noisy neighbors. They told us we needed to increase the size of our VMs.