It's a bit complex and almost always in the midst of a structural change due to new learnings[0][1][2], so describing it in text will be a bit difficult both to read and write. I am planning to do a nice write-up with diagrams once I get myself a proper platform to broadcast myself (which it's taken me ten years to continue to fail to do, so don't hold your breath).
Firstly, as fbdab103 said:
This right here is the key to backups: triage your data to understand what is truly important.
Know your data, know how often it changes, know its level of importance to you, know what duration of loss you're willing to incur (or are likely to incur based on hardware cost limitations).
Hardware:
- Primary NAS with large internal storage and matched external USB storage
- Secondary NAS with same storage size as Primary NAS
- 2x Servers running Proxmox, each with internal storage enough for their VMs data
- Spare older hardware from past upgrades (not live, but as redundancy in case of disaster)
Software:
- Proxmox
- Syncthing
- rsync
- NAS software (QNAP, Synology, TrueNAS, or whatever else)
Configuration:
I have a central store of documents on the Primary NAS, protected by username / password access. Each family member has their own directory.
Mobiles backup photos and videos to a temporary / throw away location on the Primary NAS via a Syncthing docker instance hosted on one of the VMs. This destination is then backed up (twice?) daily using a couple of rsync scripts to separate photos and videos into a permanent location on the Primary NAS.
The secondary NAS has a daily wake up and backup of these 'home directories', photos and videos to both its internal and external storage.
Each VM is hosted on the Proxmox server's internal storage (I used to have all the VM disks hosted on the NAS and connected to Proxmox via ISCSI, but the centralisation to the NAS became an issue when the NAS was lost - see New Learnings 1[0]).
Each Proxmox Server has a weekly scheduled backup of all the VMs to both the Primary and Secondary NAS's, and at least 4 historic backups, in addition to the current, are kept, which Proxmox is configured to manage.
(This is actually against what I said earlier about the Secondary NAS only 'fetching' backups - that's how it used to do it, but this went against the 'purity' of having a direct backup raher than a copy of a backup, which is its own interesting topic: a direct backup is more trustworthy than a copy of a backup because there's less opportunity for data corruption, so the theory goes).
The primary NAS has a weekly backup of these VM backups from its internal storage to external storage.
The secondary NAS has a weekly wake up and backup of these VM backups from its internal storage to external storage.
The secondary NAS also has a weekly wake up and backup of other media stored on the primary NAS, which gets copied to both internal and external storage.
The external storage of either or both of the NAS's can be disconnected and stored elsewhere if I go on holiday, then retrieved and reconnected upon return.
In addition, I do ad-hoc backups to older HDDs and SSDs of different tiers of data, where Family Photos and Videos are probably the most import sentimentally, and financial / household-management documents are the most important practically. These get spread amongst friends and family and are poorly documented (which I need to improve, lest they find said 'spare' hard drive or USB and decide to use it for something far less important).
(I have also recently setup backups for docker instance configuration data, but they're a bit redundant given that these are encapsulated by the VM backups, but it helps me sleep easier to have them backed up separately - these are a weekly script, similarly copied weekly from Primary to Secondary NAS internal and external storage)
[0]: New learnings 1: In mid-August last year my primary NAS became unavailable suddenly and irretrievably with no notice - an incident I'll be documenting in detail some time in the future
[1]: New learnings 2: Due to some power issues late December 2022 I'm rearranging the setup to ride through such things a bit better. This doesn't directly to the backups, but some of the hardware / configuration setup has changed, which has a bearing on the backup configuration.
[2]: New learnings 3: Stagger some of the backups in case of source getting corrupted / encrypted maliciously (may require a manual backup for the final step for a 'known good' backup to a more-secure destination)*