Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Spaces data and backups deleted by hackers

codespaces.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Code Spaces data and backups deleted by hackers

#31

Post by lugg »

opendais wrote: post #21

Seems to be down so cache: http://webcache.googleusercontent.com/search?q=cache:qpjW4k2... "We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time. Code Spaces : Is Down! Dear Customers, On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparen…

Thanks for the cache just getting to this.

Edited because someone didn't like my original tone. Was a bit rushed to be honest.

Few things seem off about this:

- Offsite backups were also deleted, I don't think they had offsite backups, or at least backups you could legitimately say were "off site."

- EC2 has two factor auth, why you wouldn't use this for your business I don't know. [1]

- Corresponding with extortionist is a really dumb move. It would be better time spent locking things down - contacting amazon directly to get an account lock / getting your ducks in a row.

[1] http://aws.amazon.com/iam/details/mfa/

Re: Code Spaces data and backups deleted by hackers

#32

Post by toomuchtodo »

bagosm wrote: post #9

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

DevOps here.

There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access).

Show of hands how many people here are doing it this way.

Re: Code Spaces data and backups deleted by hackers

#33

Post by judk »

maxlybbert wrote: post #28
bagosm wrote: post #9

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else. As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.

Two-factor authentication is a second basket.

Sending a copy to Glacier is a second basket.

Does Amazon not have 30-day undelete for bulk storage? Seems crazy.

Re: Code Spaces data and backups deleted by hackers

#34

Post by jacquesm »

lugg wrote: post #31
opendais wrote: post #21

Seems to be down so cache: http://webcache.googleusercontent.com/search?q=cache:qpjW4k2... "We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time. Code Spaces : Is Down! Dear Customers, On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparen…

Thanks for the cache just getting to this. Edited because someone didn't like my original tone. Was a bit rushed to be honest. Few things seem off about this: - Offsite backups were also deleted, I don't think they had offsite backups, or at least backups you could legitimately say were "off site." - EC2 has two factor auth, why you wouldn't use this for your business I don't know. [1] - Corresponding with extortioni…

There is something about this whole story that feels weird, I can't name it but it is as if this isn't the whole story.

Re: Code Spaces data and backups deleted by hackers

#35

Post by jacquesm »

toomuchtodo wrote: post #30
jacquesm wrote: post #25

Is it possible to 'lock' your amazon control panel to a specific set of IP addresses? In the payment world it is a fairly common feature to use a block-by-default strategy for such crucial controls. Hosting your project management and your sources with other companies always did feel strange to me. I can see how it works well for open source project and git (after all, every repo is a complete copy) but to host the m…

> So much for that I guess, if it is spinning and online it is not a backup. True, although with S3, you can make backups very difficult to remove: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelet... "If a bucket's versioning configuration is MFA Delete enabled, the bucket owner must include the x-amz-mfa request header in requests to permanently delete an object version or change the versioning state of…

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data.

And when you've made your backup you take it offline so that no matter what you can get back in business.

Re: Code Spaces data and backups deleted by hackers

#36

Post by toomuchtodo »

jacquesm wrote: post #35
toomuchtodo wrote: post #30

Earlier quoted context omitted.

> So much for that I guess, if it is spinning and online it is not a backup. True, although with S3, you can make backups very difficult to remove: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelet... "If a bucket's versioning configuration is MFA Delete enabled, the bucket owner must include the x-amz-mfa request header in requests to permanently delete an object version or change the versioning state of…

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data. And when you've made your backup you take it offline so that no matter what you can get back in business.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options.

If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes.

How much are you willing to spend to have AWS Export send you a physical SATA drive nightly?

Re: Code Spaces data and backups deleted by hackers

#37

Post by jacquesm »

toomuchtodo wrote: post #36
jacquesm wrote: post #35

Earlier quoted context omitted.

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data. And when you've made your backup you take it offline so that no matter what you can get back in business.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options. If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes. How much are you willing to sp…

That's one of the reasons I never used anything like AWS. (Another is that for my kind of usage they're just too expensive)

Re: Code Spaces data and backups deleted by hackers

#38

Post by tylerlh »

Kequc wrote: post #24
invertedohm wrote: post #12

Leaving aside the obviously deficient sysadmin work here: the timeline of the story doesn't add up. I can only hope this explanation is not accurate. You find notes in your AWS control panel saying you should contact some Hotmail address. OK. So the first thing you do is reach out to that address and take the time to communicate intricate extortion details? Only after that you think maybe it's a good idea to start ch…

I feel that a lot of people here are being unnecessarily harsh. It was all a bit of a silly mistake in hindsight but Code Spaces was a very new service I'm not even certain it had secured funding yet. The timeline looks to me like email address shows up. Check email address. Email address contains extortion details. Try to change passwords. Hacker gets in again and again while deleting stuff. Cannot get rid of hacker…

They have existed for a number of years[0]. Even still, being a new service is no excuse for such poor handling of OpSec.

[0]: https://twitter.com/CodeSpaces/status/265757401368637440

Re: Code Spaces data and backups deleted by hackers

#40

Post by EwanToo »

toomuchtodo wrote: post #36
jacquesm wrote: post #35

Earlier quoted context omitted.

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data. And when you've made your backup you take it offline so that no matter what you can get back in business.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options. If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes. How much are you willing to sp…

At its simplest, you should be able to backup to another Amazon S3 setup, that's completely isolated, belonging to a separate account.

Backups should be initiated from a production account access key where "Create" access has been granted, but all the storage and maintenance by another AWS account with it's own access key.

However, I'm not sure that's technically feasible at the moment, without quite a lot of manual scripting

Post reply on HN