Don’t use DigitalOcean for anything serious. My team and i frequently support DigitalOcean customers with issues like crashed applications failed builds and in many cases the challenge is how they (DigitalOcean) provide resource allocations for droplets. 1cpu != 1cpu which is vital for vm to perform at its best. AWS and GCP are better. i’d argue that most IaaS platforms are better than DIgitalOcean. You get what you pay for. i’m not a customer.
Why the “Digital Ocean killed my company” incident scares the hell out of me
171–180 of 185 posts
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#172'before switching to AWS completely'
why rely on a single party, you are putting all your eggs in a single basket again.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#173Earlier quoted context omitted.
They did have backups, only problem backups was also on DO. Backing up petabyte of data even on a Gbit connection is slow, and also cost more then a pennies if you for example back it up on AWS. Linus tech tips have a video on youtube explaining the problem.
It's not really a "backup" if it's just on the same server/data center though. Moving data from one drive to another in the same machine, or from one machine to another in the same building, provides fault tolerance should the main drive/machine die, that's about it. That's true whether the computer(s) is your own or someone else's.
It's all about threat model. And yes "our cloud provider accidentally deletes both our data and our backups" should be in the list.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#174I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…
> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…
My negative record is "getting access to the server" until "hard disk died" is 12h. And no, this isn't 1 out of 1 server.
I'd never again run anything business-critical on Hetzner metal. I'm not even running my irc bouncer there, tbh. The cloud stuff I'm really happy though, higher uptimes than on most physical servers there.
Oh, and remind me again of the time where we regularly called them because of outages because our monitoring was better than theirs and the customer support people hadn't yet learned of the network outage...
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#175Arguably Raisup should have been better prepared for this eventuality. But , I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before. I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get h…
> Or maybe the guy was an idiot. But possibly entertain the thought that maybe his situation wasn't exactly comparable to your eh? I have never encountered a business that does not require backups. Not having a DR site as early in the game as they were might be excusable, but even then suggesting they get one is sound advice. It's an additional expense but it has a cheaper implementation cost while your infrastructur…
> I have never encountered a business that does not require backups.
Me neither. But unfortunately the most I've encountered don't either have truly full backups (missing stuff), actually working backup process (some issue with backup media, process, etc.) or any idea how to actually restore if something does happen.
Almost no one does drills to restore backups to some test system.
I can't talk about the other cases, but I can tell about one small startup I joined a long ago. The worst one. They did backups on a 3-disk RAID-5 server... with one disk broken and one with SMART warnings. Their backup process also failed to backup anything with too long path names, so in reality almost half of the data was actually missing! There was also some unicode issue that lost files with characters above 128 ASCII.
My first days went to just actually ensuring their data is backed up...
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#176I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…
> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…
[deleted]
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#177Earlier quoted context omitted.
Are you using managed databases then?
I am. Mostly for the backup and other DB administration that comes with it. Also use Spaces (DO's S3 equivalent) for any assets that aren't packaged with the code like user uploads.
Future wish: all VPSes have managed databases, object hosting and centralized logs as standard so as not to rely on local disk at all. There is a streaming backup protocol, which allows any database and queue vendors to stream to any location, and acts as a peer for fully synchronous replication. If your DO Postgres, Kafka and Mongo instances are killed, you have a Bakstream cluster in Linode which lost no data and allows you to restore in Vultr.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#178Earlier quoted context omitted.
I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role. They want to automate that all away as much as possible. The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy ta…
> nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role It's not about money. I worked at a unicorn that paid very high wages for support staff AND allowed them to work remotely and asynchronously from anywhere in the world. If you lived in Southeast Asia or Eastern Europe you'd make more than a local doctor just…
I think it is about money... but also skilled leadership who understand how a good support organization works, how to find people and retain them.
I suspect though that due to the general trend to look at support as a "cost" most skilled leadership has moved on or just settled for poor support practices and etc.
Interestingly in my experience support teams that operate outside the home country of the company OFTEN have massive turnover issues, more than say domestic (wherever domestic is). There's a gap there that just never seems to fill in completely.
There is also something to be said for managing support in the sense that you don't have to talk to every 10,000 customers ;)
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#179Earlier quoted context omitted.
I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does. I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones…
> Also, why would they want to? One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money. Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with…
This happened to me. Rather than have an email address that only works in one country, I stopped using the email address.
It was a pain.
Re: Why the “Digital Ocean killed my company” incident scares the hell out of me
#180Earlier quoted context omitted.
There is Google Apps which is a paid service but most people gravitate to the free stuff.
Paying for the product isn't the issue, the company in the DO debacle was happy to pay - support is the issue. I pay Amazon to send me stuff, but I'd honestly be happy to pay them $5 to have my emails to support read & replied to by a person that doesn't have to rely on a low tier auto translate. I understand that support is expensive, and especially for ad financed or cheap products, having an agent look into someth…
> the company in the DO debacle was happy to pay
Best as I can tell they are a early stage startup surviving off startup credits from DO. Also $5 doesn't go far towards support costs. After a minute or two troubleshooting and they are already losing money.