Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup: Efficient file backup system based on the git packfile format

github.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Bup: Efficient file backup system based on the git packfile format

#21

Post by t0mas88 »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

We use a dedicated server. It's a fairly basic machine with a single Xeon CPU and 32GB ram and a lot of drive-bays, which I think doesn't cost much. It has at the moment 16 x 3TB drives in it in RAID 50. RAID 50 is not super on performance, but fast enough to saturate gigabit in sequential operations (which backups are). So it has 32 TB of useful storage for a price of around € 700 per month (leased server at a high-end hosting company, so could be much cheaper if you buy it yourself or use a cheaper provider). Per TB that's around € 21.5 per month. Although our reasons for doing this were not based on the prices of storage in the cloud, it was based on having the data on our own machine with disk encryption and only a connection to our internal management network and not the public internet.

Re: Bup: Efficient file backup system based on the git packfile format

#22

Post by rakoo »

rlpb wrote: post #17
gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

I wrote ddar, which is basically this but solves that particular problem, by using something other than the git packfile format. http://www.synctus.com/ddar and http://github.com/basak/ddar It's recently been made available on Homebrew, too.

I don't see a link to the source code on your page, is it just me ?

Re: Bup: Efficient file backup system based on the git packfile format

#23

Post by tobias2014 »

Another backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup".

So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date`

With zfSnap (https://github.com/graudeejs/zfSnap) you can tell how long incremental backups/snapshots are kept, "rsync && zfSnap -d -a 1w backup"

You can take advantage of the /backup/.zfs/snapshot directory to access all snapshots, built-in compression and possible data deduplication.

If you also have ZFS on the remote host, you can use zfs send and zfs receive to transfer the snapshot directly to the backup server, instead of using rsync for the diff.

Re: Bup: Efficient file backup system based on the git packfile format

#24

Post by tekacs »

tobias2014 wrote: post #23

Another backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup". So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date` With zfSnap ( https://github.com/graudeejs/zfSnap ) you can tell how long incremental backups/snap…

rsync.net is an example of a host that does something like this (with daily snapshots)

    $> ssh rsyncnet ls .zfs/snapshot
    daily_2014-02-09
    daily_2014-02-10
    daily_2014-02-11
    daily_2014-02-12
    daily_2014-02-13
    daily_2014-02-14
    daily_2014-02-15
They allow you to customise the length of time for which these snapshots are kept, too (IIRC) (at the cost to you of the incremental extra storage)

Re: Bup: Efficient file backup system based on the git packfile format

#26

Post by tekacs »

apenwarr wrote: post #9
mattdeboard wrote: post #3

I'm assuming you're not the author. But just in case the author wanders by: How did you decide which parts to write in C?

It's not too hard actually. A line of python is roughly 80x slower than a line of C (no exaggeration). But a typical line of python does a lot more than a typical line of C. So things you can do with a "loose" loop (like once per 64k block) is usually ok in python. Things you have to do with a "tight" loop (like once per byte) need to be in C. I once did a presentation about python performance optimization lessons fr…

I'd be curious to know if your stance on PyPy has changed at all since 2011 (if indeed it's something that you've taken any new long looks at since) given their progress in that time.

I know that I would humbly submit at the least that my position has moved to believing that PyPy is a viable option for high-speed code (albeit in substantial part due to better interaction with C, nowadays).

Re: Bup: Efficient file backup system based on the git packfile format

#27

Post by nathancahill »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

AWS is never the cheapest option. AWS is great when you're quickly scaling up and down, but the flexibility comes at a (high) cost.

For backups, you're dealing with a relatively consistent or predictable amount of data. Buy the appropriate dedicated server for your needs.

Re: Bup: Efficient file backup system based on the git packfile format

#28

Post by jedbrown »

tobias2014 wrote: post #23

Another backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup". So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date` With zfSnap ( https://github.com/graudeejs/zfSnap ) you can tell how long incremental backups/snap…

I do the same with btrfs. I actually tried bup first, but had a series of problems (see mailing list) and switched to btrfs snapshots. My main disk is also btrfs so I send incremental snapshots ("btrfs send -p"; faster than rsync and I can keep using the machine without making the backed-up state inconsistent), but the rsync method is fine for other source file systems.

Re: Bup: Efficient file backup system based on the git packfile format

#29

Post by rlpb »

rakoo wrote: post #22
rlpb wrote: post #17

Earlier quoted context omitted.

I wrote ddar, which is basically this but solves that particular problem, by using something other than the git packfile format. http://www.synctus.com/ddar and http://github.com/basak/ddar It's recently been made available on Homebrew, too.

I don't see a link to the source code on your page, is it just me ?

See my github link for the latest source. The other page predates me uploading to github, so there's only a tarball source download from there. I should move everything over to github now, really.
Post reply on HN