Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time-Machine-style backup with rsync

github.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Time-Machine-style backup with rsync

#31

Post by whalesalad »

This is really cool. For me, most of my data is "in the cloud" now. Important and frequently-accessed project files are in Dropbox, code is on Github, and most of my music is either streamed from the web in Spotify or my actual library is stored/streamed via iTunes Match. Because of this, I actually don't have a considerable amount of data to keep backed-up, and a lightweight non-time-machine solution like this looks perfect. I tried rsync before but never got into a solid routine.

I've become obsessed with the "12factor" app approach everywhere in my digital life, so that if a device ever disappeared, was stolen, or died, I could get a replacement fully operational without any problems. Like an app-server dying, just launch a new one and it will bootstrap itself.

I'm kinda crazy with my new "homelab" and started it off with an old 2U Poweredge I got on eBay for about $200. It's cheaper than a Synology/Drobo, has room for 6 drives, and the dual quad-core processors + 16GB of RAM is pretty cool too. It's running FreeNAS right now in a VM with a 3TB ZFS pool. I've created an AFP share that appears to my mac as Time Machine and over my gigabit-network it does a pretty fast backup. I really like the FreeNAS software. It's open-source, runs on FreeBSD, and the UI/admin tool is built in Django.

Re: Time-Machine-style backup with rsync

#32

Post by scottlu2 »

Link-backup does this as well, plus it knows how to build hard links to old backups even when directory structure or filenames change, effectively de-dup support. It does this by building a content addressable index on the destination filesystem that backup trees hard-link against.

http://www.scottlu.com/Content/Link-Backup.html

Re: Time-Machine-style backup with rsync

#33

Post by laurent123456 »

ta_tmachine wrote: post #6

When will this trend of "like time machine" backup software going to stop ? Time Machine, as can be seen here http://www.apple.com/support/timemachine/ time machine is tightly integrated in the os and provides a self-defining interface and user experience. This github page is for a wrapper shell script around rsync, which is not like time machine. This has been going on for a while now, see Timevault ( https://wiki.u…

Time Machine is a great "enable and forget about it" solution, however it has some limitations too. For example, it can only backup to a drive, not to a folder within the drive. Also, in my case, I wanted to backup the Users folder of my Windows Bootcamp partition but it cannot be done. Excluding files during the backup is also not possible. The nice thing about a small bash script to handle all this is that it can be easily customized to your needs.

Re: Time-Machine-style backup with rsync

#34

Post by aaronbee »

gizmo686 wrote: post #27
quesera wrote: post #22

The key feature of Time Machine is hard links to directories -- which is only possible on modern HFS+ (and rsync doesn't even try). Some people like the UI too, of course. Without hard linked directories, a full --link-dest backup of a decent sized disk, with zero file changes from the previous pass, can easily consume 100MB (and take 45 minutes to perform). This disk consumption might seem insignificant, today, but…

What is the benefit of hard-linking directories relative to soft-linking them?

The problem with soft-links is that if the underlying file/directory is deleted, you are screwed. For example if you have 100 full-machine backups and want to free some space so you decide to delete every other one, you have to be careful that none of the backups you are keeping, have soft-links to files in the backups you are deleting.

With hard-links the underlying data is not deleted until all hard-links are deleted, so you can delete any individual backup directory without losing data in any other backup directory.

A soft-link is like a pointer in C whereas a hard-link is like a C++ shared_ptr, ie. reference counted.

Re: Time-Machine-style backup with rsync

#35

Post by vvhn »

quesera wrote: post #22

The key feature of Time Machine is hard links to directories -- which is only possible on modern HFS+ (and rsync doesn't even try). Some people like the UI too, of course. Without hard linked directories, a full --link-dest backup of a decent sized disk, with zero file changes from the previous pass, can easily consume 100MB (and take 45 minutes to perform). This disk consumption might seem insignificant, today, but…

the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.

Re: Time-Machine-style backup with rsync

#36

Post by wtallis »

vvhn wrote: post #35
quesera wrote: post #22

The key feature of Time Machine is hard links to directories -- which is only possible on modern HFS+ (and rsync doesn't even try). Some people like the UI too, of course. Without hard linked directories, a full --link-dest backup of a decent sized disk, with zero file changes from the previous pass, can easily consume 100MB (and take 45 minutes to perform). This disk consumption might seem insignificant, today, but…

the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Re: Time-Machine-style backup with rsync

#37

Post by UntitledNo4 »

icarus_drowning wrote: post #24
pudquick wrote: post #21

Earlier quoted context omitted.

I would guess something like this: http://www.youtube.com/watch?v=RDPzVdohrck#t=1m13s Oh, and integration with the OS X recovery partition / OS reinstallation mechanism that allows you to point to a Time Machine backup as the recovery point for your re-installation.

The integration with OS reinstall works very well, and is pretty seamless from the user's perspective. I used to do two backups-- a local TM backup and a separate cloud backup, but I found it was actually easier to just use Automator to up mount my TM volume once a day, image it, and send that up to the cloud. When my TM volume failed last year, I just pulled the latest image and put it on a replacement drive, and I…

Could you please share how you did that in Automaton? I'd love to do that to my backups.

Re: Time-Machine-style backup with rsync

#38

Post by oofabz »

wtallis wrote: post #36
vvhn wrote: post #35

Earlier quoted context omitted.

the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental.

Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS.

I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrained environment than ZFS.

Re: Time-Machine-style backup with rsync

#39

Post by XorNot »

ta_tmachine wrote: post #7
bello wrote: post #5

If you're looking for file snapshots and versioning, I've found Back In Time ( http://backintime.le-web.org/ ) to be awesome.

Better yet, try zfs.

ZFS solves the hard-link problem spectacularly well (and adds a whole bunch of data-integrity verification on top of that) with snapshots.

What I don't like about that solution is it can't dedupe (ZFS dedupe just ultimately doesn't work very well). Hence my interest (and if anyone checks my post history, my constant spruiking of) bup - which does efficient dedupe and output of git pack-files. Stick that on a ZFS volume with snapshots, and you've got block-level checksummed, versioned and deduplicated backups.

What it's all missing of course, is a pleasant interface to use it with (one which doesn't fallback to the thing I see way too often in a lot of these scripts "don't worry, we're just going to stat your entire filesystem every 20 minutes).

Re: Time-Machine-style backup with rsync

#40

Post by porker »

wtallis wrote: post #36
vvhn wrote: post #35

Earlier quoted context omitted.

the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

(How) would AuFS fit into this? I've only come across it with Docker.io, but it sounds like it handles incremental backups?
Post reply on HN