Live data from Hacker News

Kopia: Fast and secure open-source backup software

kopia.io

111–120 of 138 posts

Re: Kopia: Fast and secure open-source backup software

#111
post #106
post #28

Earlier quoted context omitted.

Can you compare vs eg Duplicati? Do you (sorry, but just checking) repeatedly test backups? Eg pull monthly and bit verify that they're correct? Are you aware of anyone testing in this way? Thanks so much!

Not the previous poster, and I don't use Kopia, but after reading Kopia features and docs, they seem to be in par. I use Duplicati quite extensively for personal backups, and didn't really have any issues. Duplicati has a web interface, so with a proper authentication in place, you can use it to remotely monitor and manage backups. Duplicati doesn't keep a local cache. It uses SQLite files for file meta data, but not…

Hmm. I'm asking because I've had some trouble with Duplicati. I use it on a laptop, and it does not like being interrupted during a backup. It also doesn't fail the backup, which would be fine; instead, it gets jammed on files, particularly the large (multigb sqlite) file it generates to track state. It remains jammed, even once network is restored, and measures upload speeds at single-digit bytes per second. I end up having to force kill it after multiple abort requests fail to stop the jammed backup, and there's multiple warnings that this can corrupt data / you shouldn't kill the process...

So anyway, I'm looking for alternatives.

Duplicati also, somewhat annoyingly, nails 100% cpu for a while during backup which spins up the fans and gets my laptop very hot. I've been meaning to see if there's a simple way to modify the code to prevent this, but I'm very unfamiliar with C#.

Re: Kopia: Fast and secure open-source backup software

#112
post #32

The state of backup tech is surprisingly bad, and runs OS deep. Even with modern solutions like restic, you have no guarantee of a consistent backup[0] unless you pair it with a filesystem that supports snapshots like btrfs, ZFS, etc, which basically no one other than power users are even aware of. Interestingly, Windows ships snapshot support by default[1], but you need admin privileges to use it. Also, it's unclear…

FreeBSD had a pretty decent option in the base system two decades ago - FFS snapshots and a stock backup tool that would use them automatically with minimal effort, dump(8). Just chuck `-L` at it and your backups are consistent.

Now of course it's all about ZFS, so there's at least snapshots paired with replication - but the story for anything else is still pretty bad, with you having to put all the fiddly pieces together. I'm sure some people taught their backup tool about their special named backup snapshots sprinkled about in `.zfs/snapshot` directories, but given the fiddly nature of it I'm also sure most people just ended up YOLOing raw directories, temporal-smearing be damned.

I know I did!

I finally got around to fixing that last year with zfsnapr[1]. `zfsnapr mount /mnt/backup` and there's a snapshot of the system - all datasets, mounted recursively - ready for whatever backup tool of the year is.

I'm kind of disappointed in mentioning it over on the Practical ZFS forum that the response was not "why didn't you just use ", but "I can see why that might be useful".

Well, yes, it makes backups actually work.

> Also, it's unclear to me what happens if you attempt a snapshot in the middle of something like a database transaction or even a basic file write. Seems likely that the snapshot would still be corrupted

A snapshot is a point-in-time image of the filesystem at a given point. Any ACID database worth the name will roll back the in-flight transaction just like they would if you issued it a `kill -9`.

For other file writes, that's really down to whether or not such interruptions were considered by the writer. You may well have half-written files in your snapshot, with the file contents as they were in between two write() calls. Ideally this will only be in the form of temporary files, prior to their rename() over the data they're replacing.

For everything else - well, you have more than one snapshot backed up, right?

1: https://github.com/Freaky/zfsnapr

Re: Kopia: Fast and secure open-source backup software

#113

Kopia is great, though it's worth noting for folks on Linux: non-UTF-8 paths aren't stored correctly [1] and xattrs aren't stored [2]. While most folks probably won't care about the former, the latter can could cause issues (eg. losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux). [1] https://github.com/kopia/kopia/issues/1764 [2] https://github.com/kopia/k…

My backups includes photos and files form the late nineties when I used dos/windows. At some point, their encoding got messed up and many file names are now not UTF-8.

I can’t be bothered to fix the names (it’s not realistically a problem), but if backup software can’t handle them, then it can’t handle backup up my data.

Re: Kopia: Fast and secure open-source backup software

#114
post #66
post #25

Replaced borg by kopia a while ago, I have "kopia snapshot create /" on most of the machines I manage, just works.

Why did you switch?

Mainly to get a real server and get many machines backuped in the same deduplicated repository without having to schedule one backup at a time.

kopia also cleanly support non Linux platforms like MacOS and Windows. It has a UI too but borg is getting some now too.

Re: Kopia: Fast and secure open-source backup software

#115

Kopia is great, though it's worth noting for folks on Linux: non-UTF-8 paths aren't stored correctly [1] and xattrs aren't stored [2]. While most folks probably won't care about the former, the latter can could cause issues (eg. losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux). [1] https://github.com/kopia/kopia/issues/1764 [2] https://github.com/kopia/k…

> losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux

True, but one should get into the practice of creating persistent policies rather than ad-hoc `chcon`'s all the time, so that a `restorecon -FR /` always restores all SELinux labels to the intended state, then the biggest hassle is just booting into initramfs shell to mount rootfs and `touch /.autorelabel`

Re: Kopia: Fast and secure open-source backup software

#116
I have been using Kopia since June 2021 since I first posted it on HN [1]

My use cases are very basic, but it just quite works all the time. My total backup is roughly 1TB with more than million files at multiple locations like local, remote SFTP and Amazon S3.

I like being able to extract the selected data(directory/file) from snapshot without restoring the whole snapshot.

de-duplcaition is icing on top.

The developer Jarek is very responsive on Kopia Slack as well.

[1] https://news.ycombinator.com/item?id=27471945

Re: Kopia: Fast and secure open-source backup software

#117

Can someone please help decide what is the "best" backup software? - Restic ( https://restic.net/ ) - Borg backup ( https://www.borgbackup.org/ ) - Duplicati ( https://www.duplicati.com/ ) - Kopia ( https://kopia.io/ ) - Duplicay ( https://duplicacy.com/ ) - Duplicity ( https://duplicity.us/ )

Personally I use borg with BorgTUI (https://github.com/dpbriggs/borgtui) to schedule backups and manage sources/repositories. I'm quite pleased with the simplicity of it compared to some of the other solutions.

Re: Kopia: Fast and secure open-source backup software

#120
post #87
post #63

We are trying to use it for large backups of a production item, and it has not been a complete smooth ride all along. We have many files (millions) and lots of churn over ~80Tb total. Kopia has exhibited some issues: - takes about 120GB (!) of ram to perform regular maintenance & takes about 5hrs to do so. There are ideas floating around to cherry pick the large inefficiencies in the GC code but it’s yet to be worked…

If you're looking for ZFS backed why not rsync.net? Am I dumb for just doing some rclone+rsync.net?

Not really dumb. I do use them too but with Borgbackup on the top (since they support it natively).

I found Borgmatic ( https://torsion.org/borgmatic/ ) to be the best way to run my backups. It takes care of everything from pruning to verifying the checksum etc... and it integrates with some monitoring (like cronitor).

So Borgmatic + rsync.net is the best combo

Post reply on HN