Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code Spaces data and backups deleted by hackers

codespaces.com

Post reply on HN
Print view View original

51–60 of 85 posts

Re: Code Spaces data and backups deleted by hackers

#51

Post by mentat »

toomuchtodo wrote: post #32
bagosm wrote: post #9

All I can say is this: If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

DevOps here. There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access). Show of hands how many people here are doing it this way.

Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...

Re: Code Spaces data and backups deleted by hackers

#52

Post by __david__ »

toomuchtodo wrote: post #36
jacquesm wrote: post #35

Earlier quoted context omitted.

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data. And when you've made your backup you take it offline so that no matter what you can get back in business.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options. If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes. How much are you willing to sp…

> Where would these backups go?

Why not to a big disk sitting on a computer in your house?

The server doesn't push to it, the home computer pulls from the server (via a cron job or something). That way an attacked can come in from somewhere else. It doesn't have to be perfect, and it shouldn't be your only backup. But having something like that is great for catastrophes like this.

Re: Code Spaces data and backups deleted by hackers

#53

Post by __david__ »

pjc50 wrote: post #3

That's extraordinary .. but it does drive home the distinction between "backup" and "online second copy of your data". Proper backups should be offline when not in use.

> Proper backups should be offline when not in use.

I'm not sure I'd go that far, but I'd say that backups on the same service as the data aren't a good idea. It would smart of use a different cloud provider for the backups and (depending on the size of the operation), even a disk sitting in someone's house somewhere, just in case.

Re: Code Spaces data and backups deleted by hackers

#55

Post by twistedpair »

__david__ wrote: post #53
pjc50 wrote: post #3

That's extraordinary .. but it does drive home the distinction between "backup" and "online second copy of your data". Proper backups should be offline when not in use.

> Proper backups should be offline when not in use. I'm not sure I'd go that far, but I'd say that backups on the same service as the data aren't a good idea. It would smart of use a different cloud provider for the backups and (depending on the size of the operation), even a disk sitting in someone's house somewhere, just in case.

That's why I have a Mini on the bookshelf that SSH's into an EC2 instance (IP restricted) during a 1 hour window each night when the router is set to open and copies that day's PGP'd backups over. Then it goes back to sleep to wake on schedule the following night. Not the same as a tape in a vault, but it's my last resort.

Re: Code Spaces data and backups deleted by hackers

#56

Post by twistedpair »

Globz wrote: post #11

How the hell would you let the faith of your entire enterprise data in the hand of Amazon? Offsite/offline backup are mandatory especially when you deal with this kind of data.

Real companies don't. I know a major video provider in Cambridge that replicates everything to S3 and Google Cloud storage for this very reason. Now, neither has ever gone down for them, but if you've got to stay up, you have a resilient architecture.

Re: Code Spaces data and backups deleted by hackers

#58

Post by opendais »

__david__ wrote: post #52
toomuchtodo wrote: post #36

Earlier quoted context omitted.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options. If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes. How much are you willing to sp…

> Where would these backups go? Why not to a big disk sitting on a computer in your house? The server doesn't push to it, the home computer pulls from the server (via a cron job or something). That way an attacked can come in from somewhere else. It doesn't have to be perfect, and it shouldn't be your only backup. But having something like that is great for catastrophes like this.

> Why not to a big disk sitting on a computer in your house?

I used to do this to save the $6 and backup locally. Unfortunately, it stopped being practical once the backups hit 3-4 GB since they'd interfere with my internet access in the morning.

Realistically, what I'd do, is have each founder [or just any 2 technical folk really] each setup seperate accounts at two vendors [e.g. Backupsy + Kimsufi] and have the VM pull the backup from the source. I'd keep a week's worth of backups in this way.

No one person could destroy 100% of the backups. A single breach would not destroy 100% of the backups [although it might destroy the production environment depending on permissions].

The cost for such a solution to cover like 1TB of data? $40/month x2.

If you are a funded startup and you aren't able to spend $100/month on securing your backups I'm not sure what to tell you.

Re: Code Spaces data and backups deleted by hackers

#59

Post by notwedtm »

__david__ wrote: post #52
toomuchtodo wrote: post #36

Earlier quoted context omitted.

Okay then. Where would these backups go? S3 is easy to backup to. Tarsnap is nice. Rsync.net works as well. But these are all online backup options. If you're advocating offloading to physical media, you need someone who is going to religiously do it (execute code, pull to removable flash drive/SATA dock), and the more up to date you want your backups to be, the more tedious it becomes. How much are you willing to sp…

> Where would these backups go? Why not to a big disk sitting on a computer in your house? The server doesn't push to it, the home computer pulls from the server (via a cron job or something). That way an attacked can come in from somewhere else. It doesn't have to be perfect, and it shouldn't be your only backup. But having something like that is great for catastrophes like this.

and then your house burns down, floods, is robbed.

Re: Code Spaces data and backups deleted by hackers

#60

Post by jacquesm »

notwedtm wrote: post #59
__david__ wrote: post #52

Earlier quoted context omitted.

> Where would these backups go? Why not to a big disk sitting on a computer in your house? The server doesn't push to it, the home computer pulls from the server (via a cron job or something). That way an attacked can come in from somewhere else. It doesn't have to be perfect, and it shouldn't be your only backup. But having something like that is great for catastrophes like this.

and then your house burns down, floods, is robbed.

One of the basic premises behind a solid backup strategy is that if disaster hits that it does not hit simultaneously in all places. If that does happen then I think you have different problems to contend with than trying to restore your backups.
Post reply on HN