Code Spaces data and backups deleted by hackers

"We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time. Code Spaces : Is Down!

Dear Customers,

On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.

An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address

Reaching out to the address started a chain of events that revolved arount the person trying to extort a large fee in order to resolve the DDOS.

Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.

At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.

In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.

This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs.

Data Status

All svn repositories that had the following url structure have been deleted from our live EBS's and all backups and snapshots have been deleted: https://[ACCOUNT].codesapces.com/svn/[REPONAME]

All Svn repositoies using the following url format are still available for export but all backups and snapshots have been deleted: https://svn.codespaces.com/[ACCOUNT]/[REPONAME]

All Git repositories are available for export but all backups and snapshots have been deleted

All Code Spaces machines have been deleted except some old svn nodes and one git node.

All EBS volumes containing database files have been deleted as have all snapshots and backups.

Code Spaces Status

Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.

As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us.

All that we can say at this point is how sorry we are to both our customers and to the people who make a living at Code Spaces for the chain of events that lead us here.

In order to get any remaining data exported please email us at support[at]codespaces.com with your account url and we will endeavour to process the request as soon as possible.

On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you, and ask for your understanding during this time! We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!"

In the payment world it is a fairly common feature to use a block-by-default strategy for such crucial controls.

Hosting your project management and your sources with other companies always did feel strange to me. I can see how it works well for open source project and git (after all, every repo is a complete copy) but to host the master of a subversion repo 'in the cloud' and to have your project management in the cloud feels uneasy to me.

If this or something like it would happen to github and all the github issues would be lost that would be a fairly major disaster.

You never know how solid the infrastructure and solutions chosen behind a nice looking web front are until it goes down, and this one went down hard.

Condolences to the users of codespaces.com, they are the ones who lost most in all this.

From codespaces backup page:

" Real Time Backups Backup

All your Source Code is backed up in real time, so that in the unlikely event of a system break down your data is safe.

Not only do we Back Up your data we also give you access to our backup up data via the Code Spaces Admin console so you can keep your own copies.

Whenever you make a change we make a backup."

So much for that I guess, if it is spinning and online it is not a backup.

(ops guy)