They ran a code hosting service and they had no offsite backups? Wow, just wow.
From their homepage "We offer Rock Solid, Secure .. hosting". Perhaps not so rock solid...
Code Spaces data and backups deleted by hackers
61–70 of 85 posts
Re: Code Spaces data and backups deleted by hackers
#62Reminds me of LiveJournal's dead harddrive induced closure. When will startups learn how it's done?
When did that happen? I was under the impression that LiveJournal was still around however it was mostly popular in Russia. Any links to that?
Re: Code Spaces data and backups deleted by hackers
#63Earlier quoted context omitted.
> So much for that I guess, if it is spinning and online it is not a backup. True, although with S3, you can make backups very difficult to remove: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelet... "If a bucket's versioning configuration is MFA Delete enabled, the bucket owner must include the x-amz-mfa request header in requests to permanently delete an object version or change the versioning state of…
Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data. And when you've made your backup you take it offline so that no matter what you can get back in business.
>>You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone
Incorrect.
The best practice is to have multiple backups at different locations, some online, some offline. The offline ones should not all be under your control. For example, companies in certain sectors store copies of their source code in bank vaults, updated once per year. The reason is clear: if something happens to your company (e.g. a plane crashes into your high-rise...), you can recover from an offsite copy of your business data. A company that used to have its headquarters in the Twin Towers recovered from Sept 11 in under 24 hours because they didn't say "all of our backups should be under our control."
Just because something is online does not mean it is not a backup. It can be a backup, it just should not be your only one.
Re: Code Spaces data and backups deleted by hackers
#64Saw this earlier today. That's rough. Obviously they had some problems with their architecture (backups shouldn’t be able to be deleted like that), but it's still pretty messed up. I hope they catch the guy. I won't help Code Spaces, but whoever it was deserves to be caught.
s/I/It/
It hope they catch the guy?
Re: Code Spaces data and backups deleted by hackers
#65Earlier quoted context omitted.
DevOps here. There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access). Show of hands how many people here are doing it this way.
Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...
Couldn't agree more. Everything under a single platform, no MFA, no (real) offsite backup, and on top of that they spent 12 hours corresponding with the attacker, instead of immediately calling Amazon to ask their help to shut down everything, while they still had time?
I'm sorry, but this is a succession of things not to do in terms of system operations. Probably the team never managed mission critical platforms before, and hopefully they now learned the lesson.
Re: Code Spaces data and backups deleted by hackers
#66Reminds me of LiveJournal's dead harddrive induced closure. When will startups learn how it's done?
Apparently they won't. I can't tell you the number of times I hear "everyone who's important in the startup world says 'shut up and ship' and we'll think about security & BC/DR later", where "later" usually ends up being "after a breach". If they survive.
Re: Code Spaces data and backups deleted by hackers
#67Re: Code Spaces data and backups deleted by hackers
#68Earlier quoted context omitted.
> Where would these backups go? Why not to a big disk sitting on a computer in your house? The server doesn't push to it, the home computer pulls from the server (via a cron job or something). That way an attacked can come in from somewhere else. It doesn't have to be perfect, and it shouldn't be your only backup. But having something like that is great for catastrophes like this.
> Why not to a big disk sitting on a computer in your house? I used to do this to save the $6 and backup locally. Unfortunately, it stopped being practical once the backups hit 3-4 GB since they'd interfere with my internet access in the morning. Realistically, what I'd do, is have each founder [or just any 2 technical folk really] each setup seperate accounts at two vendors [e.g. Backupsy + Kimsufi] and have the VM…
> I used to do this to save the $6 and backup locally. Unfortunately, it stopped being practical once the backups hit 3-4 GB since they'd interfere with my internet access in the morning.
3-4 GB of data each day?
I backup data from my personal laptop to digitalocean droplet ($5 month) and then during the night backup droplet (which also stores my mail and other stuff) to the disc connected to raspi in my home. Incremental backup (rdiff-backup) takes literally 5 minutes (3 mins for /home and 2 mins for mail). And amount of data slowly approaches 9GB.
Re: Code Spaces data and backups deleted by hackers
#69Earlier quoted context omitted.
Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else. As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.
Two-factor authentication is a second basket. Sending a copy to Glacier is a second basket. Does Amazon not have 30-day undelete for bulk storage? Seems crazy.
> Two-factor authentication is a second basket.
Not in my books. Recommended to be sure, but this has far too many single points of failure. To name a few:
- Software corrupts data
- Hardware corrupts data
- Social engineering bypasses 2FA
Re: Code Spaces data and backups deleted by hackers
#70When will these cloud people learn that storing a so called backup in the same place or under the same domain or root account as the original data is a copy not an actual backup.