Live data from Hacker News

DigitalOcean lost our server

murze.be

151–160 of 219 posts

Re: DigitalOcean lost our server

#151

I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.

Not to mention that service credit is more than I've ever gotten from Amazon when they've lost one of our VMs due to underlying hardware failure.

Re: DigitalOcean lost our server

#152
I'm more worried about the following post in that thread:

--------------------------------------------------------

We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way.

Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them.

Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups.

This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry.

We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit.

You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.

Re: DigitalOcean lost our server

#153
post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day.

What if I was on vacation, and didn't see this e-mail for a few days? shudder

In what world is automating the deletion of customer data a good idea?

Re: DigitalOcean lost our server

#154
Whats the point of raid if raid card failure resulted in complete loss of data? I came across an article a week or two ago that discussed various alternatives, I think ZFS with checksums, which unlike raid will not replicate corrupt data from drive nearing failure to the healthy drive.

Re: DigitalOcean lost our server

#155
post #133

Tell me if I'm doing something wrong: my two EC2 servers both use EBS including for the root device, so I'm not using any device-attached storage. If Amazon "lost" my instance, it would appear to me to be just a reboot. And yes, I do a nightly backup of databases, webroot, /etc and some other directories.

Do you have two separate AWS accounts? Is your backup outside of amazons influence?

Re: DigitalOcean lost our server

#156
post #153
post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?

> In what world is automating the deletion of customer data a good idea?

The data destruction world :)

Re: DigitalOcean lost our server

#157
I do understand the part where they terminated and killed of the VMs. I do not, at all, understand why they deleted the backups. How can the CEO sit in his chair and make that decision? If he didn't and he doesn't know, why is he still in that chair?

Even damn Netflix preserves data for 10 months. Deleting backups is making 100% sure that the customer doesn't come back to you. It's amazingly stupid.

Unless you're as smart and as big as Google, use the damn phone. Don't use the word automated at all. It's not automated if one fool made a decision and another one wrote the script.

Re: DigitalOcean lost our server

#158
If you're paying for backups and they get deleted the moment there's an issue with payment, ur doin' it rong. I don't care whatever the TOS says, they're being completely shitty to customers. Sure there should be regular off-site backups. No one is arguing against that. But the fastest restore is usually from the closest source, and that's partly why you'd pay them for it.

Re: DigitalOcean lost our server

#160
post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

I had the exact same experience! Except I got no credit.

My CC expired but I was on vacation and didn't have access to email. When I was back, the droplet and all backups and snapshots were completely removed. For the record, I was only 15 days late. Very sad. I tried my best to get the backups from the support engineer but no luck.

Post reply on HN