61–70 of 80 posts
Earlier quoted context omitted.
On disk the backups are encrypted, that means no transparent compression.
For people who haven't dealt with this - a good encryption scheme produces output which you can't tell apart from a purely random stream of bits - it has very high entropy, and is therefore not compressible.
Another one? Ten years ago I had trouble finding anything, then over time I learned of duplicity (2002?!¹), bup (2010), restic (2015), borg (2010)... all basically solving the same problem of encrypted incremental backups. The landing page doesn't mention why they made yet another solution. In the comments someone also mentions bvckup2. Is there an overview somewhere of all the different solutions? Any selling points…
I've only used kopia briefly and it had a bug where I couldn't simply read files from its GUI, so I'd take it it's still doing basic implementations but all the tools you mentioned have varying degree of reliability and performance as well.
You can easily Google around and see how restic takes huge memory and is slow to restore on larger repo, duplicity being unreliable and you'll find less of those reports for borg which I believe is the only open source tool of the kind that is reliable and doesn't stall on restoration.
I want to use multiple implementations for doing a backup towards multiple remote targets in case one implementation has a corruption bug but so far my research only shows it's better to just use borg twice to different remote targets than use another implementation that could be unreliable.
duplicacy also takes lots of memory to the point of freezing up a machine with 2GB+ free ram while taking medium sized repo that is only in the range of 50GB with many small files. (Not to mention I don't like its per machine license which can be costly if you handle many small servers.)
Backup tools are something that is so sensitive about its reliability. It must be working from the beginning as it's too late when your backup data was corrupt but you only realize that when your original data is already inaccessible as not many people do backup data integrity check frequently.
Reading on its feature list doesn't mean much but do yourself a favor and check for GitHub issues, check the forum for bug reports and maybe even Reddit and figure if they're fighting primitive bugs or not.
You should rely on a solid tool than just one of the remote backup tool, like using rsnapshot locally first. It uses rsync and hard links, which is quite hard to break.
Here are a few reports on instability for a few tools.
1. https://forum.rclone.org/t/rclone-as-destination-for-borgbac...
2. https://www.reddit.com/r/unRAID/comments/eg0zpe/duplicati_se...
3. https://forum.duplicati.com/t/is-duplicati-2-ready-for-produ...
Earlier quoted context omitted.
For people who haven't dealt with this - a good encryption scheme produces output which you can't tell apart from a purely random stream of bits - it has very high entropy, and is therefore not compressible.
Thanks for clarifying this. That must mean then: first encrypt, then compress?
Another one? Ten years ago I had trouble finding anything, then over time I learned of duplicity (2002?!¹), bup (2010), restic (2015), borg (2010)... all basically solving the same problem of encrypted incremental backups. The landing page doesn't mention why they made yet another solution. In the comments someone also mentions bvckup2. Is there an overview somewhere of all the different solutions? Any selling points…
Earlier quoted context omitted.
Thanks for clarifying this. That must mean then: first encrypt, then compress?
Compress than encrypt.
Another one? Ten years ago I had trouble finding anything, then over time I learned of duplicity (2002?!¹), bup (2010), restic (2015), borg (2010)... all basically solving the same problem of encrypted incremental backups. The landing page doesn't mention why they made yet another solution. In the comments someone also mentions bvckup2. Is there an overview somewhere of all the different solutions? Any selling points…
The only thing I need (and is sorely missing from Restic) is that the metadata be kept separate from the actual data. That way I can store the data in AWS S3 Deep Glacier at a cost of nothing per year, and still do incremental backups. Currently the architecture of Restic for instance requires all data to be quickly and cheaply accessible; which makes it impossible for this. I have terabytes of data that I'd be happy…
Would you mind sharing how much you pay for S3? I assume you’ve considered options like Backblaze B2 and Wasabi?
Normal S3 would be ~$1.1k/yr, or around half of that for the infrequent access tier, both of which are way too expensive.
Earlier quoted context omitted.
Would you mind sharing how much you pay for S3? I assume you’ve considered options like Backblaze B2 and Wasabi?
Suppose I have ~4 TB of data. If I dump it into Deep Glacier, it'll be ~$50/year (free ingress), if I ever need to retrieve the data, it's like $370. Normal S3 would be ~$1.1k/yr, or around half of that for the infrequent access tier, both of which are way too expensive.
Backblaze B2's calculator is a little more sophisticated, and putting in the numbers for an absolutely pathological usecase where you start with 4TB, download, delete and upload that same amount every month puts you at $720/year. A much less pathological use case (I think) that assumes you upload, delete and download 1TB/month puts out around $360/year.
Hetzner storage boxes offer 10TB for ~$48/month, which is $576 a year -- free ingress/egress, no hidden fees for operations or whatever else, but you do have to set up a node with minio (or use FTP, etc).
Amortized over the happy time (time where you don't need to rely on your backups) this does make sense, but I wonder what the percentages look like on that kind of metric. To be fair I haven't had to restore from backup for years so this probably makes a lot of sense. I guess there's no need to test your backups/restore either if you're using a tool like borg/restic/etc and have tested it with local hardware.
Also, what happens if you have to retrieve data twice from Glacier? You've got access to it for 24 hours so I assume you're planning on just keeping the data on some non-glacier storage medium for a while after the initial post-disaster pull?