11–20 of 98 posts
> That is a dataset which is already deduplicated via copy-on-write semantics (it was not using ZFS deduplication because you should basically never use ZFS deduplication). Can someone more experienced with ZFS say why?
Earlier quoted context omitted.
I can see how this would be theoretically possible in the same way I could see using `git filter-branch` to remove one or more commits from a code repository. But as it requires walking back up the tree to recalculate all of the commit hashes based on the new state of your files, I suspect it would be an extremely slow/expensive operation in bup's case. Someone who knows more about bup's internals can correct me if I…
Isn't this exactly what Obnam does?
There are existing discussions about the way bup pruning would have to work:
https://groups.google.com/forum/#!searchin/bup-list/prune/bu...
Sounds like they are actually making progress on it.
Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups.
All other common tools seem to fail on one of the following points
- Incremental forever (bandwidth is expensive in a lot of countries)
- Untrusted remote storage (so i can hook it up to a dodgy lowendbox VPS)
- Optional: No server-side processing needed (so i can hook it up to S3 or Dropbox)
If your backup model is based on the old' original + diff(original, v1) + diff(v1, v2).. then you're going to have a slow time restoring. rdiff-backup gets this right by reversing the incremental chain. However, as soon as you need to consolidate incremental images, you lose the possibility of encrypting the data (since encrypt(diff()) is useless from a diff perspective).
But with a hash-based backup system? All restore points take constant time to restore.
Duplicity, Duplicati 1.x, and Ahsay 5 don't support incremental-forever. Ahsay 6 supports incremental-forever at the expense of requiring trust in the server (server-side decrypt to consolidate images). Duplicati 2 attempted to move to a hash-based system but they chose to use fixed block offsets rather than checksum-based offsets, so the incremental detection is inefficient after an insert point.
IMO Attic gets everything right. There's patches for windows support on their github. I wrote a munin plugin for it.
Disclaimer: I work in the SMB backup industry.
A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…
Could you or someone explain why these fantastic sounding tools don't get a developed front-end? Or if they do why am I missing them?
The best solution I've found is ChronoSync.
Deleting old backups and the lack of encryption is what stopped me from using bup.
So they did the easy parts and skipped the hard parts.
Hardly a perfect backup.
I'm not sure a backup with all three of: delete old backups, encryption, and upload only differences even exists.
I think it might be one of those "pick any two" things, but if anyone knows of a backup with all three let me know.
Is there an easy way to have the backups encrypted at rest? That's a nice feature of Duplicity. I don't have to worry about someone hacking my backup server or borrowing my USB drive having access to my data.
some ppl have reported using an encrypted storage backend like ecryptfs to store their bup repositories in. that option shouldn't be too hard to put together.
> That is a dataset which is already deduplicated via copy-on-write semantics (it was not using ZFS deduplication because you should basically never use ZFS deduplication). Can someone more experienced with ZFS say why?
It boils down to the fact that ZFS maintains a mapping from hashes to LBNs. This allows write-time deduplication (as opposed to a scrubber that runs periodically and retroactively deduplicates already written blocks). This is somewhat memory intensive though. For smaller ZFS pools you can get away with just having lots of RAM (and with or without dedupe ZFS performs better the more RAM you have). For larger ones, you can add a SSD to act as additional disk cache.
Here's a quick description of that setup:
https://blogs.oracle.com/brendan/entry/test
Note in this example that they were already showing 128GB of RAM for a 17TB pool; the L2ARC was to augment that. In general, ZFS was designed with a much higher RAM/Disk ratio than a workstation typically has.
> That is a dataset which is already deduplicated via copy-on-write semantics (it was not using ZFS deduplication because you should basically never use ZFS deduplication). Can someone more experienced with ZFS say why?
I'm not an expert by any means, but the most cited reasons are that it requires a very big amount of ram, and it depends a lot (obviously) on the type of data.
A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…
For someone like me who isn't very technically minded (forgive me)... Could you or someone explain why these fantastic sounding tools don't get a developed front-end? Or if they do why am I missing them? The best solution I've found is ChronoSync.