Rackspace hardware failure, all data lost. Oops...

#1

Post by nixle » Wed, May 16, 2012, 4:00 PM UTC

Just a friendly reminder to not put too much trust in "the cloud", it was just our development server and we are rebuilding... but not with Rackspace. The thing is that this is the only message I have received about it. It's pretty much: "yeah, we lost all your data, good luck".

===========================

Hello,

This message is a follow-up to our previous emails. Despite the very best efforts of our engineering and datacenter operations teams, unfortunately the host machine that your cloud server xxxxx resides on was not able to be recovered. At this point, there is complete data loss on the server.

If you have taken a server image, your next step will be to build a new server from the server image. If you require the same IP address, please contact support prior to deleting the failed server from your account and we will assist you in sharing the IP to the new server. If you have not taken a server image, your next step will be to build a new server from a stock image and then upload data from your local backup.

We are extremely sorry for this inconvenience. Our Billing Team is going to process a xxxx credit for the server, equal to the monthly cost.

If you have any further questions, please contact a member of our support team by responding to this ticket, visiting us in live-chat, or by calling xxxxx or xxxxx

Best Regards, The Rackspace Cloud US Toll Free: xxxxxxxxx

======================

Re: Rackspace hardware failure, all data lost. Oops...

#2

Post by hluska » Wed, May 16, 2012, 4:09 PM UTC

No backups, hey? That's rough my friend and I feel bad for you. Best of luck recovering and thanks for the reminder; I'm going to make sure that my backup scripts work!!!

Re: Rackspace hardware failure, all data lost. Oops...

#3

Post by facorreia » Wed, May 16, 2012, 4:30 PM UTC

You can outsource your hardware, but you can't outsource your responsibility. Backups and redundancy are your responsibility, not your service provider's.

Re: Rackspace hardware failure, all data lost. Oops...

#4

Post by kaolinite » Wed, May 16, 2012, 4:33 PM UTC

facorreia wrote: post #3

You can outsource your hardware, but you can't outsource your responsibility. Backups and redundancy are your responsibility, not your service provider's.

..unless you pay for automated backups.

Re: Rackspace hardware failure, all data lost. Oops...

#5

Post by freehunter » Wed, May 16, 2012, 4:46 PM UTC

facorreia wrote: post #3

You can outsource your hardware, but you can't outsource your responsibility. Backups and redundancy are your responsibility, not your service provider's.

Another thing of note is your service agreement. Check what responsibilities and restitution are present in the contract. If you need them to be liable for damages beyond downtime, make sure it says that in the contract. If you need them to failover, make sure it says that in the contract. If it doesn't, well... it's pretty much: "yeah, we lost all your data, good luck".

Re: Rackspace hardware failure, all data lost. Oops...

#6

Post by toomuchtodo » Wed, May 16, 2012, 5:04 PM UTC

freehunter wrote: post #5

facorreia wrote: post #3

You can outsource your hardware, but you can't outsource your responsibility. Backups and redundancy are your responsibility, not your service provider's.

Another thing of note is your service agreement. Check what responsibilities and restitution are present in the contract. If you need them to be liable for damages beyond downtime, make sure it says that in the contract. If you need them to failover, make sure it says that in the contract. If it doesn't, well... it's pretty much: "yeah, we lost all your data, good luck".

I've done hosting for over a decade; no provider with a brain is ever liable for anything more than what you've paid for.

Re: Rackspace hardware failure, all data lost. Oops...

#7

Post by dholowiski » Wed, May 16, 2012, 5:28 PM UTC

This all depends... were you paying for a managed server that they were supposed to be backing up? If so, massive fail by rackspace. But if you were supposed to be in charge of the backups, then don't blame rackspace.

Re: Rackspace hardware failure, all data lost. Oops...

#8

Post by freehunter » Wed, May 16, 2012, 7:08 PM UTC

toomuchtodo wrote: post #6

freehunter wrote: post #5

Earlier quoted context omitted.

Another thing of note is your service agreement. Check what responsibilities and restitution are present in the contract. If you need them to be liable for damages beyond downtime, make sure it says that in the contract. If you need them to failover, make sure it says that in the contract. If it doesn't, well... it's pretty much: "yeah, we lost all your data, good luck".

I've done hosting for over a decade; no provider with a brain is ever liable for anything more than what you've paid for.

The company I work for has cloud hosting contracts which state if the service goes down, they are liable for a certain percentage of lost revenue due to the outage. This comes with a 100% uptime guarantee, including several failovers. We pay dearly for it, but if you need something specific, you'll likely be able to negotiate a contract based on what you need. If it's not in the contract, you won't get it was my point.

Re: Rackspace hardware failure, all data lost. Oops...

#9

Post by toomuchtodo » Wed, May 16, 2012, 7:45 PM UTC

freehunter wrote: post #8

toomuchtodo wrote: post #6

Earlier quoted context omitted.

I've done hosting for over a decade; no provider with a brain is ever liable for anything more than what you've paid for.

The company I work for has cloud hosting contracts which state if the service goes down, they are liable for a certain percentage of lost revenue due to the outage. This comes with a 100% uptime guarantee, including several failovers. We pay dearly for it, but if you need something specific, you'll likely be able to negotiate a contract based on what you need. If it's not in the contract, you won't get it was my poin…

Fair enough; but you're still going to limit your liability. Assume your customer spends $10K/month with you; or $120K/year. Would you ever commit to compensating them $1MM+ in the event of an outage? Of course not. It would be business suicide.

If you want that level of uptime, you build out your own place (i.e. Amazon, Facebook, Google). If you're not big enough to build your own, you live with the possibility of downtime.

Re: Rackspace hardware failure, all data lost. Oops...

#10

Post by jgeorge » Wed, May 16, 2012, 9:06 PM UTC

Rage level depends on what you're paying for. I have a server at Rackspace, it's my job to ensure backups, not theirs. That's the service level I pay for. They offer a backup service through Cloud Files that puts your backup on a different server. About once a month (which is all I need for the little usage my server gets) I log in, click the button to do a Cloud Files backup, and I'm done. Costs me about 60 cents a month for the file storage, pretty cheap for an off-server backup. If the server holding my instance fails, it's a couple of clicks and about 10min to restore it back.

I hate it for anyone who loses data, but if you're not paying them for a backup service, it's an expensive lesson to learn.