41–50 of 68 posts
Where can I read about the best solutions for backups? For nerdy people
Two of the better howtos that we have seen for borg are [3][4]. [4] is geared toward OpenBSD users.
[1] https://borgbackup.readthedocs.io/en/stable/
[2] https://www.stavros.io/posts/holy-grail-backups/
[3] https://jstaf.github.io/2018/03/12/backups-with-borg-rsync.h...
I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…
My internet upload speed is bad so I do want something like that. I would also like to be able to "stage" a backup: figure out what needs to be transmitted and then create the data files that need to be transmitted without actually immediately transmitting it. That would allow me to do things like backup my laptop to another computer in my house that can upload the files over my slow connection overnight when my lapt…
I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…
My internet upload speed is bad so I do want something like that. I would also like to be able to "stage" a backup: figure out what needs to be transmitted and then create the data files that need to be transmitted without actually immediately transmitting it. That would allow me to do things like backup my laptop to another computer in my house that can upload the files over my slow connection overnight when my lapt…
Where can I read about the best solutions for backups? For nerdy people
When talking about backups for nerds, https://www.rsync.net/ always deserves a mention.
Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.
Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.
Earlier quoted context omitted.
Rclone will do exactly what you want, upload to S3 and the --max-duration will stop new transfer from starting after a given duration. There are also throttle options for bandwidth. I use that combined with Node-Red and a smart plug on my monitors, if monitor power draw exceeds a threshold then the upload throttle is changed via the rclone API.
Ok, that looks like a nice tool. Thanks!
I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…
Rclone will do exactly what you want, upload to S3 and the --max-duration will stop new transfer from starting after a given duration. There are also throttle options for bandwidth. I use that combined with Node-Red and a smart plug on my monitors, if monitor power draw exceeds a threshold then the upload throttle is changed via the rclone API.
Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.
Restic’s repository format is simple and well documented, which is important for long term data recovery (and fixes in case changes occur in the repo). The crypto is from a good source, and well regarded. Multithreaded, fast, nice and clean output.
ZFS is a file system, and has serious limitation when used as a backup tool. It needs a ZFS backend, ruling out almost any provider (basically self host your ZFS system, which is costly and error prone). It needs more RAM than Borg and restic. And I personally feel uncomfortable with native encryption in ZFS til some time. Lower level system encryption is probably not what you want in backups.
One feature I miss from these tools (other than ZFS): error correction. They could use a Reed Solomon code or similar and add parities in case there is an accidental change in the repository.
Any thoughts on how this compares to duplicacy?