Malicious hackers completely destroy flight sim site • Hacker News Board

Re: Malicious hackers completely destroy flight sim site

#11

Post by mustpax » Fri, May 15, 2009, 3:35 AM UTC

ars wrote: post #7

0xdefec8 wrote: post #3

not sure if I'm more annoyed by the dumbass crackers or webmasters...

How was the webmaster a "dumbass"? They had backups, but the hacker (yes I know it should be cracker - too bad - that battle is lost), took it out.

I don't want to sound callous, but just duplicating your data once does not constitute a sound backup strategy. You need snapshots and failover redundancy. It's good to have RAID 1 style redundancy for high availability, but if you or someone malicious writes bad data to your primary, you've hosed your backup as well.

They had some backups yes, but there is great room for improvement here.

Re: Malicious hackers completely destroy flight sim site

#12

Post by cperciva » Fri, May 15, 2009, 5:48 AM UTC

ars wrote: post #6

mcav wrote: post #2

Obligatory reminder: Back up your system. Now. Automate your backup process. That is all.

They did backup - but the hacker (attacker) killed the backup server too. How do you handle that? I mean, your server has to be able to write to the backup server - so the attacker can also. It would need to be some sort of append only filesystem, and who does that? I use ssh to copy the tar to a different server - remotely distant. Which I thought was enough, for fires in the datacenter, or going out of business wit…

How do you handle that? I mean, your server has to be able to write to the backup server - so the attacker can also.

With tarsnap you can create a write-only key file (that is, a key file which only has the authentication and encryption keys for creating archives); if you do this, you can run unattended backups from a server and someone who breaks into the server won't be able to read or delete your backups. (Tarsnap doesn't have any concept of modifying existing archives, so being able to create an archive doesn't allow you to overwrite existing data.)

Re: Malicious hackers completely destroy flight sim site

#13

Post by oomkiller » Fri, May 15, 2009, 8:38 AM UTC

Sounds like they used two servers, and the other was probably a backup server, maybe even a db slave, designed to run the entire web app in case the other was down. This probably lead to it being compromised in the same way the main server was.

There are many good solutions mentioned to this problem, whether it be backing up to physical media once a month, or using tarsnap. I feel though that this could have been easily prevented by running a dedicated backup server, instead of one that was probably vulnerable in the same way as the main server was. In my experiences, vandals usually aren't the most brilliant bunch and usually won't go deeper into your others servers unless it's related to the original exploit, or possibly easier. Of course this could have been more than vandalism, maybe a personal vendetta or something.

Re: Malicious hackers completely destroy flight sim site

#14

Post by badger7 » Fri, May 15, 2009, 9:36 AM UTC

The site's absence from the time the content was deleted until the time it would have taken to restore a backup is the fault of the dirty, stinking crackers. From that point onwards, it is the fault of the person who decided on the backup scheme as, evidently, it was insufficient.

If I don't wear a seatbelt and am thrown from the car in a crash, that is my fault. The crash itself and injuries I would have sustained anyway might be someone else's fault, but anything resulting from the difference between wearing a seatbelt and not wearing a seatbelt is my fault. I'm negligent from the instant I put myself in a position where I could conceivably have a crash without having put a seatbelt in place.

Re: Malicious hackers completely destroy flight sim site

#15

Post by cookiecaper » Fri, May 15, 2009, 1:50 PM UTC

ars wrote: post #6

mcav wrote: post #2

Obligatory reminder: Back up your system. Now. Automate your backup process. That is all.

They did backup - but the hacker (attacker) killed the backup server too. How do you handle that? I mean, your server has to be able to write to the backup server - so the attacker can also. It would need to be some sort of append only filesystem, and who does that? I use ssh to copy the tar to a different server - remotely distant. Which I thought was enough, for fires in the datacenter, or going out of business wit…

It's about redundancy -- a backup on one backup server is better than nothing, but if you care about your data, you should be a little more careful than just storing it on two different machines. Store it on lots of different machines, store it on optical media and USB keys, store it in the swarm if possible, etc. People who think that having one tarball on one machine is sufficient have really not thought things through.