Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bup – towards the perfect backup

wrouesnel.github.io

Post reply on HN
Print view View original

91–98 of 98 posts

Re: Bup – towards the perfect backup

#91

Post by rakoo »

rlpb wrote: post #89
rakoo wrote: post #88

Earlier quoted context omitted.

It's true that git's pack files are made for being rewritten, but bup doesn't do that. Every new run will create a new pack along with its .idx (which means that some packs may be quasi-empty) and the size of packs is capped at 1GB (Giga, not Gibi). The real struggle of bup is how to know whether a hash is already stored, and how to know it screaming fast . It could be interesting to compare bup style and standard sq…

> I don't know what's the median size of your objects but if it's I can't remember the exact number off the top of my head, but I designed the average size of each object to be much bigger - more like 64-256M than kilobytes. IMHO this works far better for backups. So I just use the filesystem to store the blobs, which I think works better.

Doesn't that alleviate the benefit of deduplication, if you're working on multi megabytes objects ? You'll end up copying a lot, unless I'm missing something obvious...

Re: Bup – towards the perfect backup

#92

Post by rlpb »

rakoo wrote: post #91
rlpb wrote: post #89

Earlier quoted context omitted.

> I don't know what's the median size of your objects but if it's I can't remember the exact number off the top of my head, but I designed the average size of each object to be much bigger - more like 64-256M than kilobytes. IMHO this works far better for backups. So I just use the filesystem to store the blobs, which I think works better.

Doesn't that alleviate the benefit of deduplication, if you're working on multi megabytes objects ? You'll end up copying a lot, unless I'm missing something obvious...

Most backups have very large areas of duplication. If a small file has changed, chances are that the small files around it have changed also. So de-duplicating with a larger chunk size seems to work fine in practice.

Re: Bup – towards the perfect backup

#93

Post by witten »

IgorPartola wrote: post #90
witten wrote: post #57

Earlier quoted context omitted.

Great question. I do think this should be part of what attic provides out of the box, but I still really wanted to use attic despite the fact that it doesn't include this sort of functionality. I'll try contacting the attic devs and see what they say about it.

Your wrapper is only lacking one critical feature I'd love. I am currently using rsnapshot and while its big issue is lack of encryption, it is able to run scripts on remote hosts to pull backups from them. This is a big deal to me since I can then script things like MySQL/Postgres backups, etc. on my master server, rather than having to configure each host individually. It's possible that this is a bad way to run th…

Push vs. pull for backups is an interesting philosophical issue, and there's some good discussion of it below. For many setups, pull really doesn't make sense. For example, one of the machines I backup is my personal laptop, and I back it up to a completely untrusted VPS. Therefore I want to be able to encrypt locally and push that encrypted data to the remote VPS. Pulling wouldn't work here, because then I'd have to hand the keys to my laptop to the VPS.

The scenario you're describing, however, sounds like the opposite in terms of trust. And in that case pull may make sense. However it doesn't sound like attic itself natively supports that sort of config. I could envision a sort of hybrid approach where the local machine encrypts to a local attic repository, and then the remote backup server pulls a copy of it. There's nothing stopping you from setting that up, either with attic as-is or with this wrapper script.

Re: Bup – towards the perfect backup

#94

Post by wglb »

kijin wrote: post #32
mappu wrote: post #14

A shoutout for attic https://attic-backup.org/ Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups. All other common tools seem to fail on one of the following points - In…

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH…

Which is why tarsnap says about your keys: STORE THIS FILE SOMEWHERE SAFE! Copy it to a different system, put it onto a USB disk, give it to a friend, print it out (it is printable text) and store it in a bank vault — there are lots of ways to keep it safe, but pick one and do it.

Re: Bup – towards the perfect backup

#95

Post by keehun »

cpach wrote: post #9
keehun wrote: post #2

This seems like a fantastic tool, and I would love to try this out. And, it's free! My personal obstacle in using a tool like bup is the back-up space. I could definitely use this for on-site/external storage devices, but I also like to keep online/cloud copies. I currently use CrashPlan for that which affords me unlimited space. If CrashPlan would let me use their cloud with bup, wow, I would switch in a heartbeat.…

You might want to have a look at Tarsnap: https://www.tarsnap.com/efficiency.html

Aware of Tarsnap and looks very attractive. I'd rather pay a flat-rate out because I store a lot of video (Musician) and other large files. I have about 750GB pre-deduplication stored. That's a lot more $ when I go to Tarsnap.

Re: Bup – towards the perfect backup

#96

Post by glandium »

jlebar wrote: post #68

Adding a plug for git-annex. https://git-annex.branchable.com/ git annex is for more than just backups. In particular, it lets you store files on multiple machines and retrieve them at will. This lets you do backups to e.g. S3, but it also lets you e.g. store your mp3 collection on your NAS and then easily copy some files to your laptop before leaving on a trip. Any changes you make while you're offline can be sync'e…

http://git-annex.branchable.com/not/

"git-annex is not a backup system."

Re: Bup – towards the perfect backup

#97

Post by ClashTheBunny »

rsync wrote: post #61
kijin wrote: post #32

Earlier quoted context omitted.

Sorry, but "Untrusted remote storage" and "No server-side processing" are exactly the opposite of what I need. If the original box is ever compromised, I don't want the attacker to gain any access to the backup. If you use a dumb storage like S3 as your backup server, you need to store your keys on the original box, and anyone who gains control of the original box can destroy your S3 bucket as well. Ditto for any SSH…

I really try to restrain myself when a backup article pops up on HN, but there are two things you raise here that I'd like to address ... first: "Ditto for any SSH-based backup scheme that requires keys to be stored on the original box. A compromised box could also lie about checksums, silently corrupting your backups." This is a good thought - you should indeed be thinking about an attacker compromising your system…

Is it possible to be notified if a certain percent of the backup is changed? Something that would let me tell if something like 50% of the bytes or 50% of the files are different between snapshots? Just a simple 'zfs diff | wc -l | mail' in cron?

Re: Bup – towards the perfect backup

#98

Post by rlpb »

rlpb wrote: post #89
rakoo wrote: post #88

Earlier quoted context omitted.

It's true that git's pack files are made for being rewritten, but bup doesn't do that. Every new run will create a new pack along with its .idx (which means that some packs may be quasi-empty) and the size of packs is capped at 1GB (Giga, not Gibi). The real struggle of bup is how to know whether a hash is already stored, and how to know it screaming fast . It could be interesting to compare bup style and standard sq…

> I don't know what's the median size of your objects but if it's I can't remember the exact number off the top of my head, but I designed the average size of each object to be much bigger - more like 64-256M than kilobytes. IMHO this works far better for backups. So I just use the filesystem to store the blobs, which I think works better.

Correction: I'd forgotten the details. Looks like I aimed for 256k, and it is this size that works well. I did consider filesystem performance when I chose the size, intending flat file blob storage here.
Post reply on HN