FreeBSD: Home NAS, part 1 – configuring ZFS mirror (RAID1)

This works, regardless of if you have ram errors or not.

I will say that the reported error rate of 5 bit errors per 8 GB per hour in 8% of installed RAM seems incredibly high compared to my experience running on a fleet of about one to three thousand machines with 64-768 GB of ECC RAM. Based on that rate, assuming a thousand machines with 64 GB ram each, we should have been seeing about 3000 bit errors per hour; but ECC reports were rare. Most machines went through their 3-5 year life without reporting any correctable errors. Of the small handful of machines that had errors, most of them went from no errors to a concerning amount of errors in a short time and were shut down to have their ram replaced; a few threw uncorrectable errors, most of those threw a second uncorrectable shortly thereafter and had their ram replaced; there was one or two that would do about one correctable error per day and we let those run. There was one, maybe two that were having so many correctable errors that the machine check exceptions caused operational problems that didn't make sense until the hourly ECC report came up with a huge number.

The real tricky one without ECC is that one bit error a day case... that's likely to corrupt data silently, without any other symptoms. If you have a lot of bit errors, chances are the computer will operate poorly; you'll probably end up with some corrupt data, but you'll also have a lot of crashing and hopefully run a memtest and figure it out.

I've lost work and personal data to bit rot in NAS filesystems before. Archived VM images wouldn't boot anymore after months in storage. Multiple vacation photos became colorful static part way through on disk due to a bit flip in the middle of the JPEG stream. I've had zero issues since switching to ZFS (even without ECC.)

Another huge benefit of ZFS is the copy-on-write (CoW) snapshots, which saved me many times as an IT administrator. It was effortless to restore files when users accidentally deleted them, and recovering from a cryptolocker type attack is also instant. Without CoW, snapshots are possible, but they're expensive and slow. I saw a 20-user office try to snapshots on their 30TB Windows Server NAS hoping to avoid having to revert to tape backups to recover the occasional accidentally deleted file. While hourly snapshots would have been ideal, the NAS only had room for only two snapshots, and would crawl to a halt while it created them. But ZFS's performance won't suffer if you snapshot every minute.

When it's time to backup, ZFS' send/recv capability means you only ever move the differences when backing up, and they're pre-computed so you don't have to re-index an entire volume to determine that you only need to move 124KB, making small transfers are lightning fast. Once backup completes, you have verified that the snapshot on both sides is bit-for-bit identical. While this is the essential property of a backup, most filesystems cannot guarantee it.

ZFS has become a hard requirement for any storage system I build/buy.

Solaris 11 made boot environments a mandatory part of the OS, which was an obvious choice with the transition from UFS to ZFS for the root fs. This came into Solaris development a bit before Solaris 11, so it was present in OpenSolaris and lives on in many forms of illumos.

It seems weird in 2025/2026 we are still discussing the baseline of getting a storage working.

Feels we’re spending too much time discussing the trees and not enough time getting the forest going: * we need reliable local storage * integrated backup * apps installation / management * remote access and account management * app isolation, reliable updates