Live data from Hacker News

Bup – towards the perfect backup

wrouesnel.github.io

31–40 of 98 posts

Re: Bup – towards the perfect backup

#31
post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

From: https://attic-backup.org/quickstart.html#quickstart

"Attic can initialize and access repositories on remote hosts if the host is accessible using SSH."

Fantastic. Will work perfectly here. We[1] are happy to support this just like we've supported duplicity all of these years. EDIT: appears obnam also works over plain old SSH. Can't tell about zbackup, however...

As always, email us to discuss the "HN Readers" discount.

[1] rsync.net

Re: Bup – towards the perfect backup

#32
post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need.

If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups.

Backups should be pulled from the backup box, not pushed from the original box. Pushing backups is only acceptable for consumer devices, and even then, only because we don't have a reliable way to pull data from them (due to frequently changing IP addresses, NAT, etc).

The backup box needs to be even more trustworthy than the original box, not the other way around. I'm willing to live with a significant amount of overhead, both in storage and in bandwidth, in order not to violate this principle.

The backup box, of course, could push encrypted data to untrusted storage, such as S3. But only after it has pulled from the original box. In both cases, the connection is initiated from the backup box, not the other way around. The backup box never accepts any incoming connection.

Does Attic support this kind of use case? The documentation doesn't seem to have anything to say about backing up remote files to local repositories. I don't see any reason why it won't be supported (since rsync does), but "nominally supported" is different from "optimized for that use case", and I suspect that many of the latest generation of backup tools are optimized for the opposite use case.

Re: Bup – towards the perfect backup

#33
post #32
post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH…

What about append-only remote storage? This is possible (in a kludgy way) in S3: http://stackoverflow.com/questions/10592541/amazon-s3-acl-fo...

Re: Bup – towards the perfect backup

#35
post #33
post #32

Earlier quoted context omitted.

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH…

What about append-only remote storage? This is possible (in a kludgy way) in S3: http://stackoverflow.com/questions/10592541/amazon-s3-acl-fo...

Better, but it might still expose to the attacker more data than he would otherwise have access to. For example: production box only contains data from the last 3 days, but the backup contains data from the last 12 months.

Even stricter access controls (write once, no read) might help with that. Not sure if you can do that with S3 though.

Re: Bup – towards the perfect backup

#36
post #23

Deleting old backups and the lack of encryption is what stopped me from using bup.

some ppl have already started working on this and there's been activity on the mailing list lately about this topic. however it's a dangerous feature to add in (backup tools should never screw up their storage, and this feature goes and removes things) so a lot of care is needed. the boring answer is: it's coming, and we need a lot of help for vetting patches and testing them out.

I think naming backups and archiving the old files should be separate applications - that could be generic to whatever backup tool you decide to use.

Re: Bup – towards the perfect backup

#37
post #11

Earlier quoted context omitted.

I'm not an expert by any means, but the most cited reasons are that it requires a very big amount of ram, and it depends a lot (obviously) on the type of data.

In addition to the memory requirements, I seem to recall that it works at the block level, as opposed to the file level. So you could have two of the same file, but maybe one copy is written at the start of a block and one is written in the middle of a block. Same file, different blocks, so no deduplication.

I'm pretty sure that ZFS doesn't do block-suballocation, so that scenario is not possible.

Re: Bup – towards the perfect backup

#38
I wrote a very similar tool before I knew about bup - ddar (https://github.com/basak/ddar - with more documentation at http://web.archive.org/web/20131209161307/http://www.synctus...).

Others have complained here that bup doesn't support deleting old backups. ddar doesn't have such an issue. Deleting snapshots work just fine (all other snapshots remain).

I think the underlying difference is that ddar uses sqlite to keep track of the chunks, whereas bup is tied to git's pack format, which isn't really geared towards large backups. git's pack files are expected to be rewritten, which works fine for code repositories but not for terabytes of data.

Re: Bup – towards the perfect backup

#39
post #16

Earlier quoted context omitted.

And the reason they are not in bup is that those two things are the only hard things in a backup. So they did the easy parts and skipped the hard parts. Hardly a perfect backup. I'm not sure a backup with all three of: delete old backups, encryption, and upload only differences even exists. I think it might be one of those "pick any two" things, but if anyone knows of a backup with all three let me know.

Tarsnap does all of these.

At present I need backup software, not a backup service.

How are you calculating a diff of an encrypted file? I looked through the technical documentation but if it explains it I missed it.

Post reply on HN