Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup: Efficient file backup system based on the git packfile format

github.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Bup: Efficient file backup system based on the git packfile format

#31

Post by kbar13 »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

You shouldn't think about backups as paying for $/GB. You should think of it has paying for the ability to restore correctly instantly when you want to restore. That's where the value really lies.

Re: Bup: Efficient file backup system based on the git packfile format

#32

Post by jrockway »

kimjotki2 wrote: post #30

a 'backup system' that runs on python. how oxymoron.

Seriously. I write all my critical software in assembly so that my super-fast disks and networks aren't bottlenecked by unnecessary CPU instructions! Backup software always values speed over correctness!

Re: Bup: Efficient file backup system based on the git packfile format

#33

Post by acdha »

natch wrote: post #10

How do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really…

Remember that there's a huge difference between files you need to read and write at any time with low latency and backups which happen at larger intervals and read infrequently, with looser latency demands. For backups, the service to compare is Glacier, which is about an order of magnitude cheaper.

What I'd consider is essentially the Crashplan model: P2P / external backups locally (i.e. full LAN speed) and an off-site replica which can be cheaper and slower as long as you have a high confidence that it'll be available eventually. This way normal operations are fast but if the building burns down you're covered and presumably have higher priorities than waiting for a restore to run.

Re: Bup: Efficient file backup system based on the git packfile format

#34

Post by isomorphic »

kimjotki2 wrote: post #30

a 'backup system' that runs on python. how oxymoron.

You mean like rdiff-backup, which I've been using in production against millions of files for more than half a decade without a single problem?

It may be that some languages or runtimes host consistently more reliable software than others, but I'd bet that the individual programmer, coding style and practice have more of an effect on reliability.

Re: Bup: Efficient file backup system based on the git packfile format

#35

Post by developuh »

tobias2014 wrote: post #23

Another backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup". So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date` With zfSnap ( https://github.com/graudeejs/zfSnap ) you can tell how long incremental backups/snap…

Where can I learn more about using rsync and ZFS ?

Re: Bup: Efficient file backup system based on the git packfile format

#36

Post by aristidb »

rlpb wrote: post #17
gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

I wrote ddar, which is basically this but solves that particular problem, by using something other than the git packfile format. http://www.synctus.com/ddar and http://github.com/basak/ddar It's recently been made available on Homebrew, too.

Nice. Could you add some description there how the deduplication works? I assume it uses rolling checksums to create chunk boundaries, just like bup?

Re: Bup: Efficient file backup system based on the git packfile format

#37

Post by raphinou »

rlpb wrote: post #17
gcr wrote: post #4

Bup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)

I wrote ddar, which is basically this but solves that particular problem, by using something other than the git packfile format. http://www.synctus.com/ddar and http://github.com/basak/ddar It's recently been made available on Homebrew, too.

How actively is it developed? How reliable is it currently?

Re: Bup: Efficient file backup system based on the git packfile format

#38

Post by atso »

I hear about rdiff-backup, but I think its two main drawbacks are:

* on the webpage, there is no new release since 2009.

* has no de-duplication.

I was considering moving my 5+ years old rdiff-backup system to any of those new, promising programs:

* obnam [http://liw.fi/obnam/]

* attic [https://pythonhosted.org/Attic/]

They both do automatic de-duplication, old backup deletion and remote encryption.

Re: Bup: Efficient file backup system based on the git packfile format

#39

Post by limmeau »

atso wrote: post #38

I hear about rdiff-backup, but I think its two main drawbacks are: * on the webpage, there is no new release since 2009. * has no de-duplication. I was considering moving my 5+ years old rdiff-backup system to any of those new, promising programs: * obnam [ http://liw.fi/obnam/ ] * attic [ https://pythonhosted.org/Attic/ ] They both do automatic de-duplication, old backup deletion and remote encryption.

The original rdiff-backup author went on to create duplicity. Maybe a big part of the rdiff-backup community has followed him?

I'm using obnam now.

Re: Bup: Efficient file backup system based on the git packfile format

#40

Post by brimstedt »

so, its 2014 and still people use homegrown variations of tar, rsync, git and whatnot. Or a half done solution like this, or an abandoned solution like box backup.

why on earth isnt there already a perfect cross platform open source backup program? :)

I know,i know..why dont i make one myself? because we dont need a nother half done solution :-b

Post reply on HN