Live data from Hacker News

Ask HN: How do you backup your files without depending on a third party service?

news.ycombinator.com

81–90 of 112 posts

Re: Ask HN: How do you backup your files without depending on a third party service?

#81
If you do that (just rsync every day), what happens when your primary storage gets corrupted, and you rsync the corrupted copy over your USB backup?

Much better to do some sort of incremental or differential backup.

As someone else said, borg is a good way of doing this.

Given you're using cloud storage, you could use rclone to sync (one way, from cloud to local) to a local disk, and then use borg (on your local machine) to backup the local copies to a second drive.

Of course, it's best to have this rclone+borg setup in two locations (e.g. second location can be parents' house or kids' house) so that it's very very unlikely that the borg repo(s) get corrupted due to user error or software malfunction.

Re: Ask HN: How do you backup your files without depending on a third party service?

#82
Many options here for backups, but I didn't see most of them touch upon backup integrity verification and losing information through bad backups (any of the storage devices/locations suffering from bitrot or other corruption that silently stays and gets propagated) that may not be noticed until it's too late (during restore time, which may happen quite rarely).

So, how do you know your backups are good and that what you're backing up is good?

Re: Ask HN: How do you backup your files without depending on a third party service?

#83

Synology or QNAP NAS. They collect backups from all computer in the house, encrypt them and ship the encrypted blobs to AWS Glacier. Synology in particular is a fantastic platform which does way more than a traditional NAS. It's quite polished personal cloud platform (way more so than FreeNAS) and includes hundreds of apps, even the backups into the cloud can be done in different ways. I also place a premium on the f…

+1 to Synology. Love their hardware and software.

Another advantage is that they're reasonably open (modules, plugins, etc), and the base operating system is GPL (but unfortunately not the UI / modules), so you can play with it when needed.

I had to play with the source code a few times over the years, and having the ability to understand what the operating system is doing [1][2], or be able to do some hardcore recovery through the serial console [3] is pretty remarkable for a consumer storage product.

(note: I wrote these posts 5-6 years ago, and the software has evolved a lot since them. I haven't checked the source code or used the serial console recently, so would love to hear thoughts if anyone has done it recently)

[1] https://wrgms.com/reverse-engineering-synology-openssh/

[2] https://wrgms.com/synologys-secret-telnet-password/

[3] https://wrgms.com/recovering-a-failed-synology-diskstation-d...

Re: Ask HN: How do you backup your files without depending on a third party service?

#84
post #19

Two FreeNAS servers on premises and in different locations with ZFS replication. Then use SMB, NFS, Time Machine, rsync, Syncthing, sftp or anything else you like to push data to one of the FreeNAS machines. Periodic ZFS snapshots prevent common cases of PEBCAK and ransomware. Fully redundant, encrypted & open-source solution that can saturate 10GbE!

If you have ecc memory and two locations this is probably the best option.

Re: Ask HN: How do you backup your files without depending on a third party service?

#85
As a private user, I just use either Clonezilla Live or an old version of Norton Ghost, and burn that to media (DVD-RW lately, it used to be CD-RW until that became unfeasible, and before that floppies). I prefer the RWs because that means I can use the discs from 2 or 3 images ago and recycle them.

Re: Ask HN: How do you backup your files without depending on a third party service?

#86
No exactly answering your requirement (not using 3rd party) but i use crashplan for multiple reasons:

- they support all platforms (mac, linux, windows) - agent running on your system, detects changes, encrypts data and transparently send stuff to the cloud.

- They support client-side encryption (key or password). If you lose password or key - you screwed.

- they support unlimited versioning of back up files. If you screw your PPT or DOC - you can restore previous version(s).

- they support unlimited destination size (i'm already at 0.5TB and counting)

- they keep deleted files.

- they support removable media attached to your system (which helps me greatly as I do lots of hi-res photography where each image is 50MB. I already got 5TB external drive ready to be sync-ed)

- everything is $120/yr

Disadvantage:

- above price is per device. So if you need to backup 2 separated computers - you'll need to pay twice, etc...

Conclusion: I been enough through semi-shitty geeky solutions and love crashplan because it's a no brainer to setup and forget. Of course - if they suddenly get out of business or cloud will explode or whatever - then you're screwed. But IMHO chances are - this backup will work for me better than anything i tried before.

Re: Ask HN: How do you backup your files without depending on a third party service?

#87
ZFS.

ZFS snapshots and clones are like git for your entire filesystem. ZFS replication over ssh is like rsync but more efficient. ZFS supports multiple compression methods so you can optimize for latency or compression ratio. Zpools let you add new disks and grow your filesystem without having to mess around reformatting. Zpools support various redundancy configurations so you can survive disk failures. It also supports checksumming so you can monitor and prevent data corruption.

https://www.freebsd.org/doc/handbook/zfs-zfs.html

Re: Ask HN: How do you backup your files without depending on a third party service?

#88
post #59
post #8

I have a 4TB Hard Drive attached to an OpenWRT router (Rasberry Pi's are also a popular choice). The router exposes a locked down SSH port (SFTP only) through a cheap $5 a month VPS. I also use Jottacloud [1] who provide unlimited (yes unlimited) storage for 7.5 EUR a month. Based in Norway, they keep an up to date warrant canary. Client side, I backup to both of these services using Duplicati [2] which offers client…

I just started experimenting with Syncthing because Dropbox no longer meets my needs. So far I'm pretty happy and impressed, it's a decentralized piece of software that Just Works (TM). My plan for backups is to spin up a VPS somewhere and have them back it up. I also love that it can use LAN; I can finally sync up my giant mp3 collection.

Syncthing is awesome. My main use of it is to share between my laptop and seedbox. Also if I want to share something with my friends in an unattended manner.

Re: Ask HN: How do you backup your files without depending on a third party service?

#89

Earlier quoted context omitted.

> Drop in a 1TB SSD for ~$200 and you're good to go. O_o ...For some reason, my mental model of SSD rates was stuck in 2014 at $0.50/GB. As I type this, my laptop is busy moving a VM image over to the NAS to free up some space on my 256 GB SSD. But no, according to https://pcpartpicker.com/trends/price/internal-hard-drive/ and https://pcpartpicker.com/products/internal-hard-drive/ , you can indeed find SSDs at $0.20/…

After my last SSD died, I'd have a hard time trusting SSD's again for backup purposes (this is probably 8-10 years ago though I suppose). It was more the way it failed - I had one or two boots where a few errors started to pop up, and then the next boot, nothing - the disk had disappeared to the BIOS' eyes and any other tools - just dead as a doornail with no easy way of getting any of the data off it. I assume they'…

8-10 years ago is a very long time, especially for what was still a relatively new, low-volume product in the consumer market.

> I assume they've improved in that regard, or are spinning platters still a safer proposition over the long term?

I don't think you need to assume..just check out what statistics are out there. Sadly, I don't know of anyone publishing numbers for consumer SSDs like Backblaze does for spinning disks.

Re: Ask HN: How do you backup your files without depending on a third party service?

#90
post #19

Two FreeNAS servers on premises and in different locations with ZFS replication. Then use SMB, NFS, Time Machine, rsync, Syncthing, sftp or anything else you like to push data to one of the FreeNAS machines. Periodic ZFS snapshots prevent common cases of PEBCAK and ransomware. Fully redundant, encrypted & open-source solution that can saturate 10GbE!

If you have ecc memory and two locations this is probably the best option.

If you don’t have a second location you can still replicate to https://www.rsync.net with all the benefits ZFS provides.

ECC is highly recommended but even a low powered CPU will do.

Post reply on HN