Time-Machine-style backup with rsync • Hacker News Board

Re: Time-Machine-style backup with rsync

#41

Post by e12e » Mon, Oct 28, 2013, 9:13 AM UTC

oofabz wrote: post #38

wtallis wrote: post #36

Earlier quoted context omitted.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental. Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS. I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrai…

Hm, how about nilfs2?

http://www.nilfs.org

Re: Time-Machine-style backup with rsync

#42

Post by torrance » Mon, Oct 28, 2013, 10:06 AM UTC

molecule wrote: post #17

This sounds a lot like rdiff-backup, which uses rsync and hard links to provide incremental backups: http://rdiff-backup.nongnu.org/features.html

rdiff-backup uses reverse diffs for its incremental backups, not hard links. This has the nice advantage of when large files change, the backup size changes only by the diff, not a whole new version of the file.

Re: Time-Machine-style backup with rsync

#43

Post by Tobu » Mon, Oct 28, 2013, 10:24 AM UTC

oofabz wrote: post #38

wtallis wrote: post #36

Earlier quoted context omitted.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental. Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS. I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrai…

Btrfs snapshots are implemented using COW-shared trees (multiple trees for data and metadata). They can't be implemented with hardlinks, that would be a very leaky abstraction.

Re: Time-Machine-style backup with rsync

#44

Post by beagle3 » Mon, Oct 28, 2013, 10:27 AM UTC

As quesera noted below, on a not-so-big modern disk with 500,000 files, the metadata can easily be in the 50-100MB range, which adds up to >1GB for metadata (even when nothing has changed) if you back up every hour.

You should, however, consider bup (https://github.com/bup/bup) - it takes less than a minute to figure out nothing is done, it deduplicates parts of files, (that is, if you have a 20GB virtual machine image, and you've changed one byte in the middle of it, then the next snapshot is going to take ~10KB, not 20GB). The older release don't keep ownership/modification time, but there's a new version pending release soon that does.

It also works well remotely (through ssh), can do an integrity check (bup fsck), redundancy (using par2; important after deduplication). And it has a fuse frontend that makes it all accessible as a file system, as well as an ftp frontend.

bup is teh awesome.

Re: Time-Machine-style backup with rsync

#45

Post by beagle3 » Mon, Oct 28, 2013, 10:28 AM UTC

oofabz wrote: post #38

wtallis wrote: post #36

Earlier quoted context omitted.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental. Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS. I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrai…

Why is block level deduplication worse?

Re: Time-Machine-style backup with rsync

#46

Post by fulafel » Mon, Oct 28, 2013, 11:07 AM UTC

wtallis wrote: post #36

vvhn wrote: post #35

Earlier quoted context omitted.

the hard linking of directories is a hack not really a feature. There's a good reason why filesystems ( including HFS+ when not being used for Time Machine) do not do it.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Why would deduplicating the metadata be important for this use case? Static amount of available inodes on some filesystems is the only reason I can think of.

Re: Time-Machine-style backup with rsync

#47

Post by unhammer » Mon, Oct 28, 2013, 11:15 AM UTC

hannibal5 wrote: post #28

molecule wrote: post #17

This sounds a lot like rdiff-backup, which uses rsync and hard links to provide incremental backups: http://rdiff-backup.nongnu.org/features.html

or rsync into zfs filesystem with snapshots.

or rsync into (optionally LUKS-encrypted) btrfs filesystem with snapshots.

Re: Time-Machine-style backup with rsync

#48

Post by icarus_drowning » Mon, Oct 28, 2013, 2:52 PM UTC

UntitledNo4 wrote: post #37

icarus_drowning wrote: post #24

Earlier quoted context omitted.

The integration with OS reinstall works very well, and is pretty seamless from the user's perspective. I used to do two backups-- a local TM backup and a separate cloud backup, but I found it was actually easier to just use Automator to up mount my TM volume once a day, image it, and send that up to the cloud. When my TM volume failed last year, I just pulled the latest image and put it on a replacement drive, and I…

Could you please share how you did that in Automaton? I'd love to do that to my backups.

I found that just running an Automator script for the "new disk image from selection" command on the root of the drive worked perfectly-- set an iCal event to run that script once a day, and you're done.

Be sure to test this to make sure it restores, but in my case it works flawlessly.

Re: Time-Machine-style backup with rsync

#49

Post by wtallis » Mon, Oct 28, 2013, 6:51 PM UTC

oofabz wrote: post #38

wtallis wrote: post #36

Earlier quoted context omitted.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Volume snapshots are a better way. The only filesystems I know of that support them are ZFS, BtrFS, and HAMMER. I'm not sure if snapshots are implemented using hard links or if they are more fundamental. Another way, which is worse, is block-level deduplication. All of the above filesystems support it, as does NTFS. I wish Apple would adopt HAMMER for Mac OS. It is BSD-licensed and more suitable for a memory-constrai…

I'm not aware of any volume snapshot system that presents the history as a plain directory tree. At best, you can use some special mount options to mount a snapshot, but you generally have to use a special tool particular to that filesystem in order to access a snapshot, and creating the snapshot always requires such a tool. There's no standard Unix way to create or access volume snapshots. Time Machine histories can be created, accessed, and analyzed entirely with standard tools except for the modified `ln`.

Re: Time-Machine-style backup with rsync

#50

Post by wtallis » Mon, Oct 28, 2013, 6:56 PM UTC

fulafel wrote: post #46

wtallis wrote: post #36

Earlier quoted context omitted.

It's a hack, but it's also the best (only?) way to deduplicate all that metadata without breaking the ability to access the history using pre-existing filesystem operations.

Why would deduplicating the metadata be important for this use case? Static amount of available inodes on some filesystems is the only reason I can think of.

Because of the aforementioned problem where it takes 100MB and the better part of an hour just to record a new snapshot when literally nothing has changed since the previous snapshot. Deduplicating gets rid of that 100MB overhead, and being able to explicitly do it with hard links instead of having to rely on the filesystem to discover the duplication on its own takes care of most of the running time overhead.