Live data from Hacker News

Time Machine-style backups with rsync (2018)

samuelhewitt.com

31–40 of 57 posts

Re: Time Machine-style backups with rsync (2018)

#31

Need to combine this with LVM or BTRFS or similar to be a true snapshot. Rsnapshot supports LVM snapshot pretty good.

Once you have btrfs you don't really need rsync anymore, its snapshot + send/receive functionality are all you need for convenient and efficient backups, using a tool like btrbk.

Re: Time Machine-style backups with rsync (2018)

#32

Earlier quoted context omitted.

> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted. Deduplication helps minimize space, but isn't it a major liability in backups? I mean, what happens when you try to restore your backups but a lone sector holding a file from way back in the past happens to not be recoverable? Doesn't it mean that no matter how frequent your backups are,…

This is bordering on paranoia. If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway. Do modern disks even have physical "sectors" anymore? Isn't it all virtual? Without dedup you're just going to backup less stuff, which is far worse.

> This is bordering on paranoia.

What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID?

> If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.

Do you honestly failed to understand that the problem is actually the "lose everything" part?

Re: Time Machine-style backups with rsync (2018)

#34
Used this technique (albeit counter-based, not time-based) successfully for backing up the 2006 World Cup site newsfeed[0] configurations/data. Had 6 rotating backups on two hosts. Never had to use them (as best I remember) but it was definitely comforting to know they were there in case of urgent rollback.

[0] Not strictly the right name but I've forgotten. Y!uk people of that era know what I mean.

Re: Time Machine-style backups with rsync (2018)

#36
post #4

If we spoke about a Linux box, one that prudently ran ZFS, or XFS on top of LVM, it would be possible to make a snapshot before the diffing and sending, so that the snapshot would be indeed point-in-time. IDK if whatever macOS uses for the filesystem supports snapshots. Otherwise, I think, restic or kopia are better for proper backups, and Syncthing for keeping a mirror copy. But the simplicity of this script in char…

> But the simplicity of this script in charming.

An important feature of backups is the ability to restore them. As much as I love restic, I have at least one backup target with hard links.

Re: Time Machine-style backups with rsync (2018)

#37
This is just half of what Time Machine does. What people are constantly missing is that Apple Time Machine is fast, as it does not need to walk through the whole filesystem to find changed files. Thanks to FSEvents, introduced in Mac OS X Leopard, it knows which directories actually contain changed files and hence usually only needs to check a small fraction of the filesystem. (Not sure if it still works that way after the switch to APFS).

This would of course also be possible on Linux (using *notify), and there are some projects which try to do this, but it's really hard to do it reliably. You might argue that this feature is less important nowadays because NVME SSDs are so fast, but still, I remember very well how astonished I was that creating a new time machine snapshot on OS X Leopard took mere seconds.

Re: Time Machine-style backups with rsync (2018)

#38
post #37

This is just half of what Time Machine does. What people are constantly missing is that Apple Time Machine is fast , as it does not need to walk through the whole filesystem to find changed files. Thanks to FSEvents, introduced in Mac OS X Leopard, it knows which directories actually contain changed files and hence usually only needs to check a small fraction of the filesystem. (Not sure if it still works that way af…

I lost all my files to Time Machine in 2008. I don't remember exactly what happened. But since then I'll take a slightly slower, observable command-line copy over sparkly magic.

Re: Time Machine-style backups with rsync (2018)

#39
post #37

This is just half of what Time Machine does. What people are constantly missing is that Apple Time Machine is fast , as it does not need to walk through the whole filesystem to find changed files. Thanks to FSEvents, introduced in Mac OS X Leopard, it knows which directories actually contain changed files and hence usually only needs to check a small fraction of the filesystem. (Not sure if it still works that way af…

No, the right way to do this on Linux and FreeBSD is to use zfs with zfs send/receive. Creating snapshots and sending them is efficient enough to use it as the underlying storage for moderately loaded databases and VMs.

They are atomic and require zero downtime. They can be encrypted and resent to other machines. Cloning whole machines from them is easy and efficient.

Re: Time Machine-style backups with rsync (2018)

#40

Earlier quoted context omitted.

In that case you are supposed to use your /other/ backup. Which you have.

Defeating the point of deduplication...

You don't have backups if you only have one "backup". Look up sysadmin's 3-2-1.
Post reply on HN