Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

171–180 of 219 posts

Re: DigitalOcean lost our server

#171

Post by rorykoehler »

mizzao wrote: post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

For what it's worth this happened to me through a client's account (client wasn't checking emails and didn't see warning mails re CC expiring). I reached out to customer support once the site went down and even though we were beyond the cut-off deadline they re-activated all the droplets. Perhaps they have updated their process in this regard. I also learned my lesson and made sure I got forwarding on all mails related to DO.

Re: DigitalOcean lost our server

#172

Post by vidyesh »

happywolf wrote: post #170
vidyesh wrote: post #164

Earlier quoted context omitted.

I really don't know to admire or criticise DO for this. On one hand horrible situation to be in, at least could have called/mailed/SMSes/notified the customer before nuking the account. On the other hand, it is a relief to know your data is secure and gone one you stop paying, no way they have a copy or keep it in archive/deep backup.

Nope, the only assurance you can get is you won't be access your data, I am not sure how one can come down with conclusion of "no way they have a copy or keep it in archive/deep backup" Having a backup and letting you to access it are two entirely different issues, and most of the time customer service staff don't know how the data management is being done by the engineering/operations team.

If you have used shared hosting you know as soon as you pay for your expired account, everything is back as it was. Never thought this was a problem until you try to test how long you can go until you pay and still restore you data.

If DO's CC can't handle data management problems they probably will connect you to their engg/ops team. So I would assume they would prefer having it resolved ( by giving the data if they had ) instead of keeping it and let user rant/post about it( which they know will probably happen ).

Re: DigitalOcean lost our server

#173

Post by [deleted] »

mizzao wrote: post #153
mizzao wrote: post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?

[deleted]

Re: DigitalOcean lost our server

#174

Post by fleitz »

xyzzy4 wrote: post #167
mizzao wrote: post #153

Earlier quoted context omitted.

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?

The world where employees care more about management satisfaction than customer satisfaction? And it's difficult to prove that bothering some customers here and there is impacting corporate revenue. Policies like this slip through the cracks.

Yes, it's digital oceans fault that people want to pay bargain basement prices and then wonder why they are getting bargain basement service.

Re: DigitalOcean lost our server

#175

Post by brobinson »

vardump wrote: post #72
twunde wrote: post #40

DO just gave a talk (~2 weeks ago) where they mentioned that they were moving to software RAID.

Sounds great, for example ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions. EMC and Netapp are of course another matter, they're great. If you can afford them.

>ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions

I'm a fan of ZFS, but this is the first time I've heard this claim. Do you have any links to articles/stories talking about it?

Re: DigitalOcean lost our server

#176

Post by speeder »

RubyPinch wrote: post #138
Fizzadar wrote: post #123

Earlier quoted context omitted.

I have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers. The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".

I'd expect to lose them. Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.

My family business website got killed once by a flood.

And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)

Re: DigitalOcean lost our server

#177

Post by joshmn »

joenathan wrote: post #169
mizzao wrote: post #152

I'm more worried about the following post in that thread: -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No bi…

I'm looking for a good host, was considering DO, but now I'm not. Any suggestions?

Needs vary, but here's a brief list of competitors:

* OVH.com — huge, great price point and performance

* Linode.com — also huge, favorite of many developers

* Vultr.com — owned by Choopa (bigger player)

* RamNode.com — doesn't scream "super modern", but reviews are great

There are many others, but those are off the top of my head that correlate with DO.

Re: DigitalOcean lost our server

#178

Post by joshmn »

chrismartin wrote: post #26

Probably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this. I agree that DO…

SolusVM — software that powers the majority of VPS range (I would love to argue how DO isn't anything cloud) provides providers with backups out-of-the-box. It's a matter of them setting it up, and usually almost always is. If it's not, well, you're probably with someone who hasn't an idea what they're doing.

Re: DigitalOcean lost our server

#179

Post by scurvy »

rubberstamp wrote: post #154

Whats the point of raid if raid card failure resulted in complete loss of data? I came across an article a week or two ago that discussed various alternatives, I think ZFS with checksums, which unlike raid will not replicate corrupt data from drive nearing failure to the healthy drive.

They should have been able to swap out the RAID card and import the configs from the disks. This is pretty common practice with LSI/Avago-based cards. Curious as to why they didn't do that. Maybe the card went really bad and started writing garbage all over the place?

Re: DigitalOcean lost our server

#180

Post by scurvy »

ekimekim wrote: post #161
Glyptodon wrote: post #20

Between the lines takeaway: hardware raid is dangerous.

Anecdotally: Every company I've worked for that uses hardware raid controllers, it feels like the controllers break about as often as one of the disks. Sure, still an improvement over no RAID, but still ridiculous compared to any decent software raid (md or ZFS).

You lose a RAID card as often as spinning metal disks? Sure that's not a wild exaggeration? Sure sounds like one. In 15+ years of doing this, I've only been a part of a RAID card swap three times. Once was just to be safe with an overly paranoid customer.

That said, I'm a fan of moving away from RAID. It's just more complexity in the face of a movement towards simpler architectures. The one benefit of a good RAID card: stupid fast, safe writes with a BBWC.

Post reply on HN