Earlier quoted context omitted.
no encryption though
You can encrypt S3 buckets/files inside your buckets. By default buckets are encrypted.
Not end-to-end encryption.
Duplicity: Encrypted bandwidth-efficient backup
51–60 of 104 posts
Re: Duplicity: Encrypted bandwidth-efficient backup
#52I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…
Could this help dedupe 20 years of ad-hoc drive dumps from Changing systems…
Re: Duplicity: Encrypted bandwidth-efficient backup
#53I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…
For those of us who prefer not to ship to the cloud, have you used Kopia Repository Server and is it any good? Does it run on Windows?
The documentation refers to files and directories. Does the software let you take a consistent, point-in-time snapshot of a whole drive (or even multiple volumes), e.g. using something like VSS? Or if you want that have you got to use other software (like Macrium Reflect) to produce a file?
Where does the client cache your encryption password/key? (or do you have to enter it each session)
Re: Duplicity: Encrypted bandwidth-efficient backup
#54Earlier quoted context omitted.
Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.
I really like restic, and am personally happy to use it via the command line. It's very fast and efficient! However, I do wish there was better tooling / wrappers around it. I'd love to be able to set something simple up on my partner's Macbook. For example, Pika Backup, and Vorta are popular UIs for Borg of which no equivalent exists for Restic, while Borgmatic seems to be a de-facto standard for profile configurati…
> For example, Pika Backup, and Vorta are popular UIs for Borg of which no equivalent exists for Restic
Have you considered?:
https://github.com/netinvent/npbackup
Or (not FOSS, but restore-compatible):
Re: Duplicity: Encrypted bandwidth-efficient backup
#55Earlier quoted context omitted.
The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/time…
It's possible to find probable duplicate files with the S3 CLI based on size and tags - I was working a script to do just that but I haven't finished it yet. Alternatively if you want exact backups of your computer you can use the --delete flag which will delete files in the bucket that aren't in the source. I agree this is not the absolute most optimized solution but it does work quite well for me and is easily exte…
If I have to choose between hacking together a bunch of shell scripts to do my deduplicated, end-to-end encrypted backups, vs using a popular open source well-tested off the shelf solution, I know which one I'm picking!
Re: Duplicity: Encrypted bandwidth-efficient backup
#56Earlier quoted context omitted.
While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner St…
> Remember folks, test your backups. Since you mention it, I am seizing the opportunity to ask: how should borg backup be tested ? Can it be automated ?
It's actually pretty simple using the check command [0]!
borg check --verify-data REPOSITORY_OR_ARCHIVE
Alternatively, I think the Vorta GUI also has a way to easily schedule it[1].
I'll add that one thing I like to do once in a blue-moon is to spin-up a VM and try to recover a few random files. While the check command checks that the data is there and theoretically recoverable, nothing really beats proving to yourself that you can, in a clean environment, recover your files.
[0] https://borgbackup.readthedocs.io/en/stable/usage/check.html
Re: Duplicity: Encrypted bandwidth-efficient backup
#57Earlier quoted context omitted.
While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner St…
> Remember folks, test your backups. Since you mention it, I am seizing the opportunity to ask: how should borg backup be tested ? Can it be automated ?
Yes, I have a bunch of scripts that allow me to pick a Borg snapshot and then do stuff with it. One such action is `borg export-tar` that just creates a TAR file
containing a full self-sufficient snapshot of your stuff.
Then just listing the files in the archive is a not-bad way to find an obvious problem. Or straight up unpacking it.
But if you're asking about a separate parity file that can be used to check and correct errors -- I haven't done that.
Re: Duplicity: Encrypted bandwidth-efficient backup
#58Earlier quoted context omitted.
Agree completely... used duplicity many years ago, but switched to Borg and never looked back. Currently doing borg-backups of quite a lot of systems, many every 6 hours, and some, like my main shell-host every 2 hours. It's quick, tiny and easy... and restores are the easiest, just mount the backup, browse the snapshot, and copy files where needed.
After Borg, I switched to Restic: https://restic.net/ AFAIK, the only difference is that Restic doesn't require Restic installed on the remote server, so you can efficiently backup to things like S3 or FTP. Other than that, both are fantastic.
Technically Borg doesn't require it either, you can backup to a local directory and then use `rclone` to upload the repo wherever.
Not practical for huge backups but it works for me as I'm backing up my machines configuration and code directories only. ~60MB, and that includes a lot of code and some data (SQL, JSON et. al.)
Re: Duplicity: Encrypted bandwidth-efficient backup
#59I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.
I've been using Restic since 2017 without issue. Tried Kopia for a while, but its backup size ballooned on me, maybe it wasn't quite ready.
Kopia supports zstd compression, I found it to be within +10% to -5% of the size of my Borg repo.
It also has extensive support for ignoring stuff and it works very well.
I still use Borg because its policy of expiring older snapshots is more useful for me, but Kopia is extremely solid and I would use it any day if I didn't care that it doesn't actually keep one monthly backup for the last 3 months as Borg does (it decides which older snapshots to keep with another algorithm; it's documented on their website).
Re: Duplicity: Encrypted bandwidth-efficient backup
#60Earlier quoted context omitted.
Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.
I really like restic, and am personally happy to use it via the command line. It's very fast and efficient! However, I do wish there was better tooling / wrappers around it. I'd love to be able to set something simple up on my partner's Macbook. For example, Pika Backup, and Vorta are popular UIs for Borg of which no equivalent exists for Restic, while Borgmatic seems to be a de-facto standard for profile configurati…
To add to this list
scheduling+browsing https://forum.restic.net/t/backrest-a-cross-platform-backup-... (Golang webUI supporting Linux and MacOS).
A number of others are findable through the community section of the forum.
Bit of a self plug, I author Backrest. The most significant challenge historically has been that restic has poor programatic interfaces but in recent revisions the JSON API (over stdout) has largely stabilized for most commands.