Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kopia: Fast and secure open-source backup software

kopia.io

Post reply on HN
Print view View original

101–110 of 138 posts

Re: Kopia: Fast and secure open-source backup software

#101

Post by KennyBlanken »

chenxiaolong wrote: post #70

Kopia is great, though it's worth noting for folks on Linux: non-UTF-8 paths aren't stored correctly [1] and xattrs aren't stored [2]. While most folks probably won't care about the former, the latter can could cause issues (eg. losing SELinux labels makes it difficult to restore a backup of the root filesystem on distros that use SELinux). [1] https://github.com/kopia/kopia/issues/1764 [2] https://github.com/kopia/k…

MacOS X makes heavy use of xattrs, too.

> Yes, there's a whole bunch of things currently not captured at the filesystem level, including setuid/gid, hardlinks, mount points, sockets, xattr, ACLs, etc.

That was three years ago and it sounds like things are only slightly better.

These people are incompetent at making backup software.

Re: Kopia: Fast and secure open-source backup software

#102

Post by mekster »

monlockandkey wrote: post #36

Can someone please help decide what is the "best" backup software? - Restic ( https://restic.net/ ) - Borg backup ( https://www.borgbackup.org/ ) - Duplicati ( https://www.duplicati.com/ ) - Kopia ( https://kopia.io/ ) - Duplicay ( https://duplicacy.com/ ) - Duplicity ( https://duplicity.us/ )

Do yourself a favor and use zfs as your primary backup, even though it means you'll have to replace your filesystem, it's just that good.

Faster than any other backup software (because it knows what's changed from the last snapshot being the filesystem itself but external backup tools always have to scan the entire directories to know what's changed), battle tested reliability with added benefit like transparent compression.

A bit of explanation on how fast it can be than external tools. (I don't work for the said service in the article or promote it.)

https://arstechnica.com/information-technology/2015/12/rsync...

Then you'll realize Borg is the one with least data corruption complaint on the internet which is good as your secondary backup.

Easily checked with, "[app name] data corruption" on Google.

And see who else lists vulnerability and corruption bugs upfront like Borg does and know the developers are forthcoming about these important issues.

https://borgbackup.readthedocs.io/en/stable/changes.html

The term "best" apparently means reliable for backup and also they don't start choking on large data sets taking huge amount of memories and roundtrip times.

They don't work against your favorite S3 compatible targets but there are services that can be targeted for those tools or just roll your own dedicated backup $5 Linux instance to avoid crying in the future.

With those 2, I don't care what other tools exist anymore.

Re: Kopia: Fast and secure open-source backup software

#103

Post by Dylan16807 »

jiggawatts wrote: post #69
xgbi wrote: post #63

We are trying to use it for large backups of a production item, and it has not been a complete smooth ride all along. We have many files (millions) and lots of churn over ~80Tb total. Kopia has exhibited some issues: - takes about 120GB (!) of ram to perform regular maintenance & takes about 5hrs to do so. There are ideas floating around to cherry pick the large inefficiencies in the GC code but it’s yet to be worked…

“ZFS on Azure via S3” is missing just an Apple technology to win the “mixing vendors randomly” bingo.

ZFS with an object store backend, is that better.

Re: Kopia: Fast and secure open-source backup software

#104

Post by tux2bsd »

AnonHP wrote: post #98

Borg 2 has been in development for nearly a year and a half [1] and may probably be released early next year, i.e., early 2024 (just a guess, seeing that even RC1 is not yet released and seems to have a lot of work to be done). Does anyone know how Borg 1.x and 2 would compare to Kopia? [1]: https://github.com/borgbackup/borg/issues/6602

I tried borg and liked it but I couldn't see how to purge 'file.big' out of prior backups. i.e. one you knew for sure you didn't need any more and would achieve disk space recovery.

Re: Kopia: Fast and secure open-source backup software

#106

Post by Ayesh »

x0x0 wrote: post #28
ntolia wrote: post #16

I've been using Kopia for my personal use and for products I have helped build at a couple of enterprise backup companies! It's also used by other open-source backup projects that focus on specific ecosystems (Velero and Kanister for Kubernetes, Corso for Microsoft 365 backup). I am obviously biased but it's pretty amazing. AMA.

Can you compare vs eg Duplicati? Do you (sorry, but just checking) repeatedly test backups? Eg pull monthly and bit verify that they're correct? Are you aware of anyone testing in this way? Thanks so much!

Not the previous poster, and I don't use Kopia, but after reading Kopia features and docs, they seem to be in par. I use Duplicati quite extensively for personal backups, and didn't really have any issues.

Duplicati has a web interface, so with a proper authentication in place, you can use it to remotely monitor and manage backups.

Duplicati doesn't keep a local cache. It uses SQLite files for file meta data, but not for the content themselves.

I like Duplicati's snapshotting mechanism. You can specify how long or how many snapshots to keep, and my anecdotal evidence is that it's archival storage-friendly. I imagine S3 and it's lifetime management rules can bring a decent and cost effective backup solution.

I'm using Google Drive 2TB plan, and I didn't see Kopia supporting Google drive out of the box.

Re: Kopia: Fast and secure open-source backup software

#107

Post by donmcronald »

mekster wrote: post #102
monlockandkey wrote: post #36

Can someone please help decide what is the "best" backup software? - Restic ( https://restic.net/ ) - Borg backup ( https://www.borgbackup.org/ ) - Duplicati ( https://www.duplicati.com/ ) - Kopia ( https://kopia.io/ ) - Duplicay ( https://duplicacy.com/ ) - Duplicity ( https://duplicity.us/ )

Do yourself a favor and use zfs as your primary backup, even though it means you'll have to replace your filesystem, it's just that good. Faster than any other backup software (because it knows what's changed from the last snapshot being the filesystem itself but external backup tools always have to scan the entire directories to know what's changed), battle tested reliability with added benefit like transparent comp…

I use ZFS + Sanoid + Syncoid locally and Borg + Borgmatic + BorgBase for offsite.

Re: Kopia: Fast and secure open-source backup software

#108

Post by donmcronald »

ntolia wrote: post #55
adobrawy wrote: post #43

Earlier quoted context omitted.

Filesystem snapshot for database is not different than database crash. Databases are designed to handle crashes well (WAL etc.).

Unfortunately, this is not true. You need to grab all the DB files (WAL, etc.) in a consistent manner. You can't grab them while writes are in progress. There are ways though. Look at what Kanister does with its recipes to get consistent DB snapshots to get a sense of the complexities need to do it "right."

I’ve always assumed a snapshot is no worse than power loss and that any decent database should handle it ok. Is that wrong?

Re: Kopia: Fast and secure open-source backup software

#109

Post by donmcronald »

MenhirMike wrote: post #46

CTRL+F "tape" "lto" - 0 results. I know that Tape Backups are not hip and sexy, but CloudNordic showed us just last month why they still matter even in 2023 and beyond, so you'd definitely want to look at an additional solution for your large servers, with a proper rotation/retention strategy (e.g., GFS). You _need_ offline backups, if you think you don't, you just got lucky for now - or have data that can be recreat…

> backing up a running Postgresql server through a file system snapshot may not be enough - there's an entire section in the documentation about backup options

https://www.postgresql.org/docs/16/backup-file.html

> An alternative file-system backup approach is to make a “consistent snapshot” of the data directory, if the file system supports that functionality (and you are willing to trust that it is implemented correctly). The typical procedure is to make a “frozen snapshot” of the volume containing the database, then copy the whole data directory (not just parts, see above) from the snapshot to a backup device, then release the frozen snapshot. This will work even while the database server is running. However, a backup created in this way saves the database files in a state as if the database server was not properly shut down; therefore, when you start the database server on the backed-up data, it will think the previous server instance crashed and will replay the WAL log. This is not a problem; just be aware of it (and be sure to include the WAL files in your backup). You can perform a CHECKPOINT before taking the snapshot to reduce recovery time.

It sounds like enough to me.

Re: Kopia: Fast and secure open-source backup software

#110

Post by Gud »

djbusby wrote: post #87
xgbi wrote: post #63

We are trying to use it for large backups of a production item, and it has not been a complete smooth ride all along. We have many files (millions) and lots of churn over ~80Tb total. Kopia has exhibited some issues: - takes about 120GB (!) of ram to perform regular maintenance & takes about 5hrs to do so. There are ideas floating around to cherry pick the large inefficiencies in the GC code but it’s yet to be worked…

If you're looking for ZFS backed why not rsync.net? Am I dumb for just doing some rclone+rsync.net?

No, you are keeping it simple.

I am using rsync to rsync.net from multiple different hosts with different configurations. I run the same command on every host running variations of *nix, no messing about with different tools needed.

Post reply on HN