Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup: Efficient file backup system based on the git packfile format

github.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Bup: Efficient file backup system based on the git packfile format

#51

Post by cbhl »

mverwijs wrote: post #44
apenwarr wrote: post #13

Earlier quoted context omitted.

My suggestion is to backup your cloud servers, which are expensive and redundant and have good uplink speeds, to home servers which are cheap and have good downlink speeds. You don't need your backup file server to be ultra-reliable or even up all the time, so the cheapest possible PC sitting on a home internet connection is a pretty good choice. That way, 3TB is just $150 or so plus your electricity, and it's not a…

I'm curious as to how you would restore that much data quickly, from a home-uplink to the cloud?

You would send the hard drive via snail-mail to Amazon.

A mass restoration is expected to be rare, so it's okay for it to be a bit more expensive.

Re: Bup: Efficient file backup system based on the git packfile format

#53

Post by drdaeman »

nisa wrote: post #14
natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

If you ignore the cloud services and rent dedicated servers you can get up to 6 TB disk space for 50$ a month. edit: 45tb for 300€/month: https://www.hetzner.de/en/hosting/produkte_rootserver/xs29

Unless you don't need any redundancy (cloud has some) that's more likely about 39TB (in RAID50, if controller supports such setup).

If so, that's still less than €8/TB/mo, which is better than most cloud storage providers offers. You also have some spare memory and CPU resources (so you could resell them for others as, for example, memcached instances) and a possibility to get a proper SLA, as a bonus.

Re: Bup: Efficient file backup system based on the git packfile format

#54

Post by sciurus »

limmeau wrote: post #39
atso wrote: post #38

I hear about rdiff-backup, but I think its two main drawbacks are: * on the webpage, there is no new release since 2009. * has no de-duplication. I was considering moving my 5+ years old rdiff-backup system to any of those new, promising programs: * obnam [ http://liw.fi/obnam/ ] * attic [ https://pythonhosted.org/Attic/ ] They both do automatic de-duplication, old backup deletion and remote encryption.

The original rdiff-backup author went on to create duplicity. Maybe a big part of the rdiff-backup community has followed him? I'm using obnam now.

For a link, http://duplicity.nongnu.org/

I backup locally using duplicity, then I ship those files off to Amazon Glacier using mt-aws.

https://github.com/vsespb/mt-aws-glacier

Re: Bup: Efficient file backup system based on the git packfile format

#55

Post by rsync »

leephillips wrote: post #52

Not really closely related, but another solution that uses git infrastructure to back up large files is git-annex: https://git-annex.branchable.com/

... and there are some tangential benefits to using git-annex as well:

http://rsync.net/products/git-annex-pricing.html

Re: Bup: Efficient file backup system based on the git packfile format

#56

Post by rsync »

tekacs wrote: post #24
tobias2014 wrote: post #23

Another backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup". So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date` With zfSnap ( https://github.com/graudeejs/zfSnap ) you can tell how long incremental backups/snap…

rsync.net is an example of a host that does something like this (with daily snapshots) $> ssh rsyncnet ls .zfs/snapshot daily_2014-02-09 daily_2014-02-10 daily_2014-02-11 daily_2014-02-12 daily_2014-02-13 daily_2014-02-14 daily_2014-02-15 They allow you to customise the length of time for which these snapshots are kept, too (IIRC) (at the cost to you of the incremental extra storage)

Just a clarification ...

All accounts get 7 days (as the parent shows) and all 1TB+ accounts get 7 days + 4 weeks.

However, if you want a custom schedule (say, 30 days, 8 weeks, and 6 months) it only costs more if the space they use exceeds your existing account.

So, if you have a 100 GB account and you use 60 GB and your fancy snapshot schedule (of which the first 7 days is always free) only uses 35 GB ... then there is no additional cost at all.

Re: Bup: Efficient file backup system based on the git packfile format

#57

Post by rsync »

res0nat0r wrote: post #45
tekacs wrote: post #24

Earlier quoted context omitted.

rsync.net is an example of a host that does something like this (with daily snapshots) $> ssh rsyncnet ls .zfs/snapshot daily_2014-02-09 daily_2014-02-10 daily_2014-02-11 daily_2014-02-12 daily_2014-02-13 daily_2014-02-14 daily_2014-02-15 They allow you to customise the length of time for which these snapshots are kept, too (IIRC) (at the cost to you of the incremental extra storage)

rsync.net looks awesome, but it is just way too expensive for me.

You should email us and ask about the "new customer HN reader discount".

Also, note that 1TB accounts are 15c per GB, per month, and 10TB accounts are 7c per GB, per month - all with no traffic or usage costs.

This compares very favorably with S3 and blows the mozy pro pricing out of the water[1].

[1] https://mozy.com/product/mozy/business

Re: Bup: Efficient file backup system based on the git packfile format

#58

Post by luxpir »

gcr wrote: post #42
jitl wrote: post #5

Earlier quoted context omitted.

Looking at the README under "Stuff that is stupid": > bup currently has no way to prune old backups Thanks for the rdiff-backup shout-out. I'm looking for a nice way to do system backups to my NAS of large VM images without having to install Crashplan. Bup and rdiff-backup both look pretty good.

Obnam is also great: http://liw.fi/obnam/ This does deduplication, but can also encrypt the deduplicated blocks and store them on (say) S3.

Obnam can also push/pull backups over SSH/SFTP. Seconded.

Re: Bup: Efficient file backup system based on the git packfile format

#60

Post by bambambazooka »

apenwarr wrote: post #11
gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

People have been actively working on a "prune" feature but it seems to never quite get finished. This would indeed be nice to have, although it's less important than you might think, given really good deduplication (which bup has). Currently bup has a very simple model - never delete anything - which is hard to screw up, so you're very unlikely to lose data.

I really have to finish that feature...
Post reply on HN