Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup – towards the perfect backup

wrouesnel.github.io

Post reply on HN
Print view View original

71–80 of 98 posts

Re: Bup – towards the perfect backup

#71

Post by cperciva »

jlebar wrote: post #67
cperciva wrote: post #43

Earlier quoted context omitted.

Tarsnap splits data into context-dependent chunks, then compares them via their hashes to figure out what new data needs to be uploaded.

This is interesting, because the size of a file's encrypted chunks now leaks information about the file's plaintext. I suppose you have some minimum chunk size, and that's one way to keep from leaking too much information as a fraction of the overall file size. But if a file is modified many times, it seems to me that you'd have to be very careful not to leak a substantial amount of data to a clever attacker. Have yo…

The chunking is done using parameters generated from a secret key, and I haven't been able to see any way for it to be computationally feasible to extract meaningful information from the resulting block sizes.

That doesn't mean that it's impossible, of course; just that it would require someone smarter than me. ;-)

Re: Bup – towards the perfect backup

#72

Post by INTPenis »

anarcat wrote: post #69
mappu wrote: post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

i was wondering how this would compare with bup. so i did a very quick benchmark: http://anarcat.koumbit.org/2014-11-18-bup-vs-attic-silly-ben... take it with a grain of salt, but i am surprised to say that attic is at least as fast as bup.

The only point that stuck in my eye was atime modification from attic but I realized that in my environment I would have to mount all the backup clients remotely on the backup server so I can get around this by mounting sshfs with noatime.

Re: Bup – towards the perfect backup

#73

Post by xorcist »

I tried some backup software (of the rdiff variety, not the amanda variety) last year when I set up a small backup server for friends and family.

Obnam and bup seemed to work mostly the way I wanted to but obnam was by far the most mature tool, so this is what I chose in the end.

On the plus side, it provides both push and pull modes. Encryption and expiration works. The minus points are no Windows support, and some horror stories about performance. Apparently it can slow to a crawl with many files. I haven't run into that problem despite hundreds of gig in the backup set, but most are large files.

On the whole it's been very stable and unobtrusive during the time I've used it, but I haven't used it in anger yet. So a careful recommendation for obnam from me.

Re: Bup – towards the perfect backup

#74

Post by uint32 »

rsync wrote: post #65
uint32 wrote: post #54

Like any good hacker I got tired of other solutions that didn't quite match my needs and made my own dropbox-like backup/sync using only rsync, ssh and encfs. https://github.com/avdd/rsyncsync Not polished, but it's working for me. - only runs on machines I control - server requirement is only rsync, ssh and coreutils - basic conflict detection - encfs --reverse to encrypt locally, store remotely - history is rsnapsh…

How does that compare to this: https://raymii.org/s/articles/Set_up_your_own_truly_secure_e...

Same principle just different mechanics and assumptions.

I can't work (very effectively) in two places at once, so I don't need robust merging, just CYA synchronisation. Using only rsync features I can do a full 2-way rsync merge and catch potential conflicts, erring on the conservative so I have reasonable confidence I don't lose any work.

Minimal workstation dependencies: only bash, encfs, rsync, ssh, coreutils/findutils and optionally atd for automation. encfs is optional, too.

Instead of dvc-autosync and XMPP I just use periodic execution. I partition my stuff into smaller/high-frequency vs larger/lower-frequency to keep this efficient. These are triggered from bash (PROMPT_COMMAND, in the background) and recursive at (atd).

The local data is unencrypted on disk from this tool's POV. I use encfs --reverse and rsync the result. To browse the history, I combine sshfs with encfs in forward mode.

Linux only because that's what I use, but it should be possible to support OSX.

All in all I'm pleased I'm able to use such minimal tooling for such a successful result.

Re: Bup – towards the perfect backup

#75

Post by williamstein »

lelutin wrote: post #50
jshb wrote: post #34

Can this new tool do incremental realtime disk image backup like Acronis True Image?

do you mean something that would automatically update the backup when a file is changed on disk? bup currently doesn't do that. but there's been some talk of using inotify or another such method of knowing exactly which files are modified when they are so that bup could instantly work on those. in theory it should be feasible, it's not implemented yet however

I had my Ph.D. student (Andrew Ohana) spend a while last summer implementing exactly this using python-inotify, since I wanted it to greatly improve the efficiency of https://cloud.sagemath.com, which makes very frequent snapshots. It's pretty solid and is on github: https://github.com/ohanar/bup/tree/bup-watch He's been busy with his actual math thesis work and teaching, so hasn't got this upstreamed into bup. It also depends on changes he made to bup to store the index using sqlite instead of some custom format.

Re: Bup – towards the perfect backup

#77

Post by rakoo »

dunham wrote: post #62
kijin wrote: post #32

Earlier quoted context omitted.

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH…

If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box I believe this isn't strictly necessary if you use asymmetric cryptography (e.g. curve25519). For a file, generate a temporary key pair, use it and the backup's public key to encrypt the file, then throw out the private key and send the encrypted file + public key to the server. Apple uses this technique to move f…

Just for the record, asymmetric cryptography is not efficient for encrypting content. What you should do is:

- Generate a temporary key

- Symmetrically encrypt with that key

- Encrypt that key with your long-term assymetric private key, and send the encrypted version along your backups.

And before you hack around your own version, I'd like to point out this is exactly what PGP (and really, any crypto scheme that involves asymmetric keys) does. So, basically, just GPG your backups.

Re: Bup – towards the perfect backup

#78

Post by konradb »

derekp7 wrote: post #58

I was wondering if someone's done a side-by-side comparison of the various newer open-source backup tools? Specifically, I'm looking for performance, compression, encryption, type of deduplication (file-level vs. block-level, and dedup between generations only vs. dedup across all files). Also, the specifics of the implementation, since some of the tools don't really explain that too well, along with any unique featu…

There's http://burp.grke.org/burp2/08results1.html

Re: Bup – towards the perfect backup

#79

Post by rakoo »

photoGrant wrote: post #15
mappu wrote: post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

For someone like me who isn't very technically minded (forgive me)... Could you or someone explain why these fantastic sounding tools don't get a developed front-end? Or if they do why am I missing them? The best solution I've found is ChronoSync.

For bup, there is something called Kup (http://kde-apps.org/content/show.php/Kup+Backup+System?conte...), but I haven't tested it.

For duplicity, there is a quite good ui in the form of deja dup (see http://www.howtogeek.com/108869/how-to-back-up-ubuntu-the-ea...). It's really nice and easy to use, and if I recall correctly it's installed by default on Ubuntu.

Re: Bup – towards the perfect backup

#80

Post by abcd_f »

beagle3 wrote: post #70

Haven't seen this mentioned - but, since bup de-duplicates chunks (and thus may take very little space - e.g., when you backup a 40GB virtual machine, each snapshots takes little more than the actual changes inside the virtual machine), every byte of the backup is actually very important and fragile, as it may be referenced from thousands of files and of snapshots. This is of course true for all dedupping and increme…

I saw "par2" and got all excited to see if lelutin re-implemented all the Galois field goodness from scratch, looked at the source and - no, bup merely spawns the par2 binary. Damn :)
Post reply on HN