Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup – An efficient backup system based on the git packfile format

bup.github.io

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Bup – An efficient backup system based on the git packfile format

#21

Post by kbenson »

gh02t wrote: post #13

Since I see various different backup utilities on here from time to time, I'm curious if anybody uses Bacula (or Bareos)? It does encryption, deduplication, scheduling, pruning etc. I use it at home and run the director and storage daemons in a jail on my FreeBSD NAS, which coordinates all of my computers together for backup regularly. I then back up the encrypted backups to Crashplan. I tried other stuff (especially…

I've sampled a few, and I've set up and used Bacula. At the time (10 years ago now), it seemed way more complex than I needed (a lot of it seemed to be centered around backing up to tape, which was not our use case). We eventually started using BackupPC, which we were really happy with, except it didn't have a native client for backing up windows, so there was a little weirdness with shadow drives, locked filed, etc. What you got in return was an open source project, a great interface, extreme versatility (do you want to back up using rsync, reverse rsync, tar-piping over SSH, or some custom mechanism?), and extreme space savings (all backup files are hard links to specific file, across multiple backups and separate backed up systems) along with reduced network usage if using rsync led to some interesting use cases we couldn't have done otherwise.

I've been out of that field for a while now through, so I'm not sure how much of that is common nowadays.

Re: Bup – An efficient backup system based on the git packfile format

#22

Post by borgel »

lucaspiller wrote: post #7

Kind of related: What do you use to backup binary files like photos? At the moment I've just got copies on old USB drives, but I'd like to put something on AWS Glacier too.

I have a layered photo backup process:

- New photos live on my laptop (in my Dropbox and covered by Backblaze)

- I manually copy photos from my laptop to a RAID array on a server elsewhere in my house when I'm done editing/processing them

- The server backs up the photos to S3

S3 is the last bastion, so to speak. I expect that normally, I pull any photos I need back off my server. But the house burns down, I go to S3.

I run a cron job on my backup server that picks up all changed files (in my photo storage dir) and pushes them to S3 with s3cmd[1]. It's a pretty 'dumb' process, but works well.

The S3 Glacier lifecycle stuff is pretty cool, so the plan is to eventually enable it to deep-archive older photos.

[1] http://s3tools.org/s3cmd

Re: Bup – An efficient backup system based on the git packfile format

#24

Post by mynegation »

wampus wrote: post #5
mynegation wrote: post #3

I do a monthly offsite backup. I use rdiff-backup for plain text and 7z with encryption for data I want to keep private, but I am on the lookout for more efficient solutions. I looked at bup, but two things stop me from using it: no encryption, and inability to delete old backups to reclaim space. To me, attic ( https://attic-backup.org/ ) looks attractive so this is what I am going to try soon.

I've used rdiff-backup daily for years, but recently needed to store backups on machines I don't control, so I gave duplicity[1] a try. It encrypts, doesn't need to be installed on the target host and is simple to use if you're already familiar with rdiff-backup and gpg. Test restores were simple, but were all first generation, so not really a good test (but there are hints you'll want to prune backups to one month's…

Thank you for the suggestion! I tried duplicity, but it uses full+differential scheme and I cannot prune old backups unless I start a new chain. 7zip with encryption does not do even that, but I went for it for the simplicity's sake for now, because duplicity did not work well with NTFS volumes.

Re: Bup – An efficient backup system based on the git packfile format

#25

Post by [deleted] »

stevekemp wrote: post #10
zeograd wrote: post #2

When I had to pick a backup system, I considered Bup until I saw that there were no way to prune old backup ( https://github.com/bup/bup/blob/master/README.md#things-that... ). This is really a stopblocker for me. Obnam ( http://obnam.org ) is a similar tool but support forgetting old generations. However, it still suffers for youth problems and tends to corrupt backup repository when pruning old data on a remote ser…

Obnam I liked using, because I love the authors attention to testing and details. But I found it started taking longer and longer and longer to run a backup - to the point that it was crazy. I switched to atic instead, which also started taking longer to run backups as more things changed, and longer still to check the archives, but it is still faster than obnam by a significant margin: https://attic-backup.org/

[deleted]

Re: Bup – An efficient backup system based on the git packfile format

#26

Post by detaro »

slight OT, but is there anything that adds transparent encryption to these backup solutions? If you're willing to live without in-file diffs, a FUSE filesystem that presents all files as GPG-encoded or something similar would be interesting, but I've never seen anything convincing like it.

Re: Bup – An efficient backup system based on the git packfile format

#27

Post by gh02t »

sliken wrote: post #20
gh02t wrote: post #13

Since I see various different backup utilities on here from time to time, I'm curious if anybody uses Bacula (or Bareos)? It does encryption, deduplication, scheduling, pruning etc. I use it at home and run the director and storage daemons in a jail on my FreeBSD NAS, which coordinates all of my computers together for backup regularly. I then back up the encrypted backups to Crashplan. I tried other stuff (especially…

Last I checked bacula doesn't do deduplication in any normal sense of the word. It allows you to run a base job and then back up diffs, that's a far cry from normal dedupe.

Oh, yeah you're right. I don't use that feature, so I didn't really know the limitations. But looking at it, base jobs, while useful, are definitely not true deduplication.

Re: Bup – An efficient backup system based on the git packfile format

#28

Post by victorhooi »

Currently, we're using Duplicity for our backups, going to Amazon S3.

I've looked a some of the newer options like Bup, Attic, Obnam etc.

However, none of them seem to support S3 as an endpoint, which is a shame. They all seem to need a full-fledged *nix server on the other end.

Does anybody know of any of these new-fangled de-dupe backup apps that works with just S3?

Re: Bup – An efficient backup system based on the git packfile format

#29

Post by sytse »

rlpb wrote: post #9
lucaspiller wrote: post #7

Kind of related: What do you use to backup binary files like photos? At the moment I've just got copies on old USB drives, but I'd like to put something on AWS Glacier too.

git-annex. It supports multiple external remotes, including USB drives, Glacier[1], bup and ddar[2]. You can keep files on any combination of the external remotes for redundancy and cost management. It keeps track of what is where, so you don't have to. The only catch is that you have to make sure to back up the (metadata-only) git repository itself also, and maintaining this backup on anything that is not a direct f…

GitLab CEO here, as you mention backing up the git repository is very important. We just announced free hosting for git-annex files and repositories on GitLab.com that might be good for your use case, see https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-... for an overview.

Re: Bup – An efficient backup system based on the git packfile format

#30

Post by rrrrtttt »

victorhooi wrote: post #28

Currently, we're using Duplicity for our backups, going to Amazon S3. I've looked a some of the newer options like Bup, Attic, Obnam etc. However, none of them seem to support S3 as an endpoint, which is a shame. They all seem to need a full-fledged *nix server on the other end. Does anybody know of any of these new-fangled de-dupe backup apps that works with just S3?

S3QL may be a good fit for your needs. It's a FUSE filesystem for S3 that supports encrypted dedup snapshots. You can do backups by rsyncing your current state into the filesystem and snapshotting it. I considered using it for my backups but ended up choosing Attic, so I can't say how well it works in practice.
Post reply on HN