Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

github.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#11

Post by reacharavindh »

firecall wrote: post #6

I'll check this out! Looks good, and havent come across it before! Despite constantly Googling for backup solutions and replacements for TimeMachine! Side note: I recently put together a TrueNAS Scale based NAS box for TimeMachine. It's running 5x 4TB drives, using ZFS RAIDZ1, and it's the best network multi-user TimeMachine destination I've ever used! (short of a large direct connected TB SSD) It's much more respons…

If you have a file system like Zfs that gives you snapshots, you would not need a tool like oP’s to make multiple copies. You can periodically do rsync(think cron job), and literally include the —delete flag such that rsync tries to replicate source and destination. The trick is to keep making cheap snapshots on ZFS such that those deletions are captured in the snapshots. When you need a file that was deleted a year ago, or as of a year ago, simply browse to the snapshot of that time, and get your data back. Zfs stores the delta only anyway, so they are quite cheap.

This is how I backup the primary storage of a huge HPC cluster - 2.5 Petabytes of research data.

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#12

Post by sgarland »

Gys wrote: post #5
pwg wrote: post #4

Earlier quoted context omitted.

The description sounds like it does largely the same job as rsnapshot ( https://rsnapshot.org/ ). What does yours do differently from rsnapshot?

Rsnapshot needs perl? Not available by default on all systems?

This needs Python 3.7. Which systems have Python 3.7 by default, but don't include Perl?

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#13

Post by sgarland »

reacharavindh wrote: post #11
firecall wrote: post #6

I'll check this out! Looks good, and havent come across it before! Despite constantly Googling for backup solutions and replacements for TimeMachine! Side note: I recently put together a TrueNAS Scale based NAS box for TimeMachine. It's running 5x 4TB drives, using ZFS RAIDZ1, and it's the best network multi-user TimeMachine destination I've ever used! (short of a large direct connected TB SSD) It's much more respons…

If you have a file system like Zfs that gives you snapshots, you would not need a tool like oP’s to make multiple copies. You can periodically do rsync(think cron job), and literally include the —delete flag such that rsync tries to replicate source and destination. The trick is to keep making cheap snapshots on ZFS such that those deletions are captured in the snapshots. When you need a file that was deleted a year…

Even better, automate snapshot management (with optional off-node sync) with sanoid/syncoid [0] (Perl) or pyznap [1] (Python).

[0]: https://github.com/jimsalterjrs/sanoid

[1]: https://github.com/yboetz/pyznap

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#14

Post by sgarland »

basnijholt wrote: post #8
locusofself wrote: post #7

How difficult do you think it would be to implement such a system in pure python? This is neat, but it still leverages a lot of shell commands, ie ssh, find, mkdir etc..

I attempted doing this initially. Everything is very doable except SSH. One would have to rely on third-party libraries to do this well.

Are you open to a single dependency [0]? Entirely native tooling is an admirable thing that I greatly appreciate, but parsing subprocess output is fraught with issues (I know, I've done this as well).

[0]: https://github.com/ParallelSSH/ssh-python

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#15

Post by pyuser583 »

basnijholt wrote: post #2

Hi Hacker News, I'm excited to share my recent project, where I took on the challenge of porting a popular but untested 600+ line Bash script to Python. The outcome is [`rsync-time-machine.py`]( https://github.com/basnijholt/rsync-time-machine.py ), a Python implementation of the [`rsync-time-backup`]( https://github.com/laurent22/rsync-time-backup ) script. It provides Time Machine-style backups using rsync and crea…

Is rsync-time-machine.py a valid module name? Or does it not matter because it’s a script?

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#16

Post by basnijholt »

ListenLinda wrote: post #9

I bet it was a fun project, but in reality backups using rsync vs restic, borg, etc., are very inefficient in speed, performance, storage, and likely other ways. After discovering these newer tools, I vowed to never put myself through the pain of rsync backups ever again.

Actually, for raw speed, rsync is much faster than any of the tools you mentioned (see e.g., https://github.com/borgbackup/borg/issues/4190). I really like a lightweight solution, where I do not even need any tool to restore backups. The tools you mentioned are great though.

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#17

Post by aborsy »

basnijholt wrote: post #16
ListenLinda wrote: post #9

I bet it was a fun project, but in reality backups using rsync vs restic, borg, etc., are very inefficient in speed, performance, storage, and likely other ways. After discovering these newer tools, I vowed to never put myself through the pain of rsync backups ever again.

Actually, for raw speed, rsync is much faster than any of the tools you mentioned (see e.g., https://github.com/borgbackup/borg/issues/4190 ). I really like a lightweight solution, where I do not even need any tool to restore backups. The tools you mentioned are great though.

Those tools do a lot more: encryption, deduplication, compression. Even then, they have been faster in my experience, due to caching.

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#18

Post by memco »

sgarland wrote: post #14
basnijholt wrote: post #8

Earlier quoted context omitted.

I attempted doing this initially. Everything is very doable except SSH. One would have to rely on third-party libraries to do this well.

Are you open to a single dependency [0]? Entirely native tooling is an admirable thing that I greatly appreciate, but parsing subprocess output is fraught with issues (I know, I've done this as well). [0]: https://github.com/ParallelSSH/ssh-python

I’ve used paramiko for this also: https://www.paramiko.org/.

Re: Python Port of 600 Line Bash Script: rsync-time-machine.py for Rsync Backups

#19

Post by indianets »

basnijholt wrote: post #16
ListenLinda wrote: post #9

I bet it was a fun project, but in reality backups using rsync vs restic, borg, etc., are very inefficient in speed, performance, storage, and likely other ways. After discovering these newer tools, I vowed to never put myself through the pain of rsync backups ever again.

Actually, for raw speed, rsync is much faster than any of the tools you mentioned (see e.g., https://github.com/borgbackup/borg/issues/4190 ). I really like a lightweight solution, where I do not even need any tool to restore backups. The tools you mentioned are great though.

Did you see the last reply on the thread you linked. The guy messed up ENV variable in borg and was doing too many account backups as new archives killing the cache when the same account was backed up next day. Borg will always be faster than rsync while doing incremental backups, but of course has a learning curve coming from the simplicity of rsync.
Post reply on HN