Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup: Efficient file backup system based on the git packfile format

github.com

Post reply on HN
Print view View original

11–20 of 63 posts

Re: Bup: Efficient file backup system based on the git packfile format

#11

Post by apenwarr »

gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

People have been actively working on a "prune" feature but it seems to never quite get finished. This would indeed be nice to have, although it's less important than you might think, given really good deduplication (which bup has). Currently bup has a very simple model - never delete anything - which is hard to screw up, so you're very unlikely to lose data.

Re: Bup: Efficient file backup system based on the git packfile format

#13

Post by apenwarr »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

My suggestion is to backup your cloud servers, which are expensive and redundant and have good uplink speeds, to home servers which are cheap and have good downlink speeds. You don't need your backup file server to be ultra-reliable or even up all the time, so the cheapest possible PC sitting on a home internet connection is a pretty good choice. That way, 3TB is just $150 or so plus your electricity, and it's not a per-month fee.

Re: Bup: Efficient file backup system based on the git packfile format

#14

Post by nisa »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

If you ignore the cloud services and rent dedicated servers you can get up to 6 TB disk space for 50$ a month.

edit: 45tb for 300€/month: https://www.hetzner.de/en/hosting/produkte_rootserver/xs29

Re: Bup: Efficient file backup system based on the git packfile format

#17

Post by rlpb »

gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

I wrote ddar, which is basically this but solves that particular problem, by using something other than the git packfile format.

http://www.synctus.com/ddar and http://github.com/basak/ddar

It's recently been made available on Homebrew, too.

Re: Bup: Efficient file backup system based on the git packfile format

#18

Post by _delirium »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

My strategy is:

1. Regularly back up "important" directories (code/, papers/, web/, etc.) to fairly safe/redundant cloud storage with incremental history. I have pretty little of this, 2. Occasionally exchange bulk but less-important backups with my brother, so we're each the other's high-latency, questionable-durability "off-site backup". No incremental dumps here, just rsync. This is where my MP3 collection, DVD rips, and similar goes.

3. Photos, which are important but also bulk, go to Flickr, which is free.

4. Don't back up stuff I can re-acquire, e.g. big public datasets I've downloaded to work on, or Debian ISOs. Also, I don't back up the OS, just my data.

There do, however, seem to be some cloud services that offer big full-disk backups for a surprisingly low flat price, e.g. http://www.backblaze.com/ is $5/mo/machine.

Re: Bup: Efficient file backup system based on the git packfile format

#19

Post by tekacs »

atmosx wrote: post #15

The most efficient backup system for operating systems I've used so far is 'tarsnap'. The only drawback is that restore is really slow.

I love Tarsnap, but S3 storage costs aren't exactly brilliant. Figuring out exactly how you wish to store keys can also be another thought, upfront (albeit one that arises from the increased security that you get 'for free').

Alternately, CrashPlan and other consumer-style services have a bad habit of using very slow, heavy, world-slowing systemwide file update scanning. :/

Having said this, a discussion of the merits and flaws of Tarsnap and similar backup services is something I'm fairly certain I've seen lengthy discussions of on similar HN posts.

(https://news.ycombinator.com/item?id=5767116 is a good source for lots of that sort of discussion)

Re: Bup: Efficient file backup system based on the git packfile format

#20

Post by sitkack »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

AWS is for startups with venture capital, ephemeral storage and compute, or someone needing to deploy a high traffic website instantly.
Post reply on HN