How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024) • Hacker News Board

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#231

Post by toast0 » Wed, Jul 08, 2026, 8:03 PM UTC

Dylan16807 wrote: post #230

toast0 wrote: post #205

Earlier quoted context omitted.

> We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC. AMD has been allowing ECC on lots of regular hardware for a long time. People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boa…

> AMD has been allowing ECC on lots of regular hardware for a long time. Often unsupported or untested by the motherboard manufacturer, because the precedent set quite a while ago has stood so strongly. > because it costs significantly more [...] and the performance specs are poor Because it's niche. If a large share of the desktop market was using ECC memory, the extra cost would have been 10-12%, and a typical kit…

> the precedent set quite a while ago has stood so strongly.

I wasn't really around for it, but I appreciate the early penny pinchers that developed fake parity ram. The IBM PC required a parity bit per byte?, but you could add an xor chip to calculate it at read and save out on the expensive ram chip.

I wouldn't mind paying a little more for ECC, what with the extra chip and a little more circuitry, but desktop ECC often starts at 50% more, which I'm way too cheap to pay for.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#232

Post by bell-cot » Wed, Jul 08, 2026, 8:24 PM UTC

bensyverson wrote: post #201

cm2187 wrote: post #111

Earlier quoted context omitted.

For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

> The odds of this happening go way up the larger your individual drives are.

Technically true.

Practically? There are several other major factors - such as the quality of your drives, whether you periodically run zfs scrubs and/or disk read tests, and how long a resilver takes to complete.

Behind those are the eternal duo - size of budget and cost of failure.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#233

Post by Fr0styMatt88 » Wed, Jul 08, 2026, 8:31 PM UTC

bensyverson wrote: post #201

cm2187 wrote: post #111

Earlier quoted context omitted.

For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

Or like me where you get two SSDs in your pool that happen to likely be from the same batch and both decide to permanently disappear at roughly the same time :(

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#234

Post by impact123 » Wed, Jul 08, 2026, 8:43 PM UTC

shark1 wrote: post #191

If you use LVM you can have a similar pool of storage with 1GB RAM or even less, instead 8GB (ZFS requirement). Caveat: LVM does not have built-in snapshots like ZFS has.

Where does that 8GB requirement come from?

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#235

Post by TacticalCoder » Wed, Jul 08, 2026, 8:44 PM UTC

Brian_K_White wrote: post #154

ocd wrote: post #104

Earlier quoted context omitted.

I have external parity for everything with rsync. I don't have any storage that isn't matched with external backups. Fortunately I've never seen any bitrot or had a drive fail on me since I tend to replace them before they do. And yeah, I could add in some extra HDDs for RAID, but this is more a personal choice because of how much I hate SATA. Filesystems like ext4 are because I value boring and rock solid stability…

rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have. Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot. Ok the tie-breaker data could…

> rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.

You're right and that's the big thing most are missing. Name your files and write your scripts accordingly I'd say.

My rsync backup'ing script first does a dry run and then serious analyses on the result of the dry run is done. If anything looks out of the ordinary, the script refuses to go ahead and warns me.

rsync silently copying bitrot to backups is the stuff nightmares are made off.

> Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.

I personally have a sizeable percentage of my files (like, say, family pictures) that have part of the checksum in the name of file:

    dsc003784-b3-79a7208c51.jpg

This means that file should have it's Blake3 checksum beginning with 79a7208c51. And these have been vetted, so in case there's a mismatch, the authoritative file is the one with the proper checksum.

And the system/disk that gave back a bogus checksum has explanations to give ; )

> Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change."

I don't think it's a myth. My script does that with the checksum as part of many filenames: if during the dry run the script notices a file with such a checksum is present but different on both systems, I know immediately that something is off.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#236

Post by tomega2134 » Wed, Jul 08, 2026, 9:07 PM UTC

solarengineer wrote: post #197

regexorcist wrote: post #194

Earlier quoted context omitted.

LVM does have snapshots.

"It is important to regularly monitor the snapshot’s storage usage. If a snapshot reaches 100% of its allocated space, it will become invalid." [1] ZFS snapshots use Copy-on-Write differently and have no such allocated space limit. Thus, we can do interesting things like snapshotting a file system after an OS is installed, and then roll back to that snapshot upon an OS upgrade failure, or even clone a new file system…

Snapshotting before OS upgrade is something that has saved my system more than once. It's not to be underestimated how much time is saved over having to restore from backup or reinstall.

Plus there is sending/receiving snapshots across systems if you want to.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#237

Post by TacticalCoder » Wed, Jul 08, 2026, 9:20 PM UTC

RossBencina wrote: post #40

I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?

> I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?

I've got a small HP Z440 workstation (small in that it's 10 years old and only single-CPU) and a 14 cores / 28 threads Xeon e5-2680 v4 in it. It's a little workhorse.

I run Proxmox (basically "Debian + ZFS out of the box + easy to spawn VMs / LXCs") on it.

And you can totally put something like FreeNAS in a VM. If you plan to really serve lots of files and need lots of perfs, you should get a tiny bit fancy and passthrough to the FreeNAS VM an entire disk controller (device passthrough to a VM in this day and age of LLMs that are there to find the relevant parts of docs for you in a split second is easy). IIRC the FreeNAS documentation itself explains how to run FreeNAS in a VM so I don't think it's "unsupported".

There's nothing wrong with running it bare either.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#238

Post by NekkoDroid » Wed, Jul 08, 2026, 10:06 PM UTC

nuker wrote: post #93

Confiks wrote: post #7

Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants). I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-fil…

> combined with LUKS, TPM keys Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack? Rabbit hole, I know, but so fascinating if you solved it all :)

> What about /boot - encrypted and tamper proof? Resists evil maid attack?

> Rabbit hole, I know, but so fascinating if you solved it all :)

Encrypting `/boot/` is the wrong thing to do. One can just replace it with something that logs the encryption key instead (and sends it off to somewhere). You actually need to verify that what you expect to be there is actually there and a encryption isn't gonna do that for you.

A reasonably secure boot chain looks roughly as follows: bootloader is checked by the UEFI (read secure-boot), bootloader chain loads a signed UKI which also uses secure boot verification (or just skip the bootloader if you only have a single kernel with no fallback), this then loads a signed dm-verity `/usr/` which brings you to the login screen. Root is just data (e.g. mounted `noexec`) and is encrypted using e.g. PCR 11 so that it only unlocks if the UKI is the one that is expected (with some other backup unlock method).

This is basically #2 of the "Design Goals" of Poetterings "Fitting everything together"[0]. It's a good read if you are looking to designing your own Linux image and are looking for some inspiration. There is ParticleOS[1] which somewhat dogfoods this kind of system, though I don't think it is anything remotely considered for a production system.

[0]: https://0pointer.net/blog/fitting-everything-together.html

[1]: https://github.com/systemd/particleos

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#239

Post by QuiEgo » Wed, Jul 08, 2026, 10:19 PM UTC

Brian_K_White wrote: post #147

dapperdrake wrote: post #88

Earlier quoted context omitted.

ATA (SATA), SCSI (SAS), and NVM (NVMe).

no. just sata & sas. There is no nvme spinning drive and flash loses bits just from sitting, it's not archival. flash is good for working not for storing.

This is true, but if you use a file system that does scrubs (and you should use such a file system for a NAS), and if you keep the NAS on 24*7, this won’t be an issue.

Re: How to Build a Minimal ZFS NAS Without Synology, QNAP, TrueNAS (2024)

#240

Post by bell-cot » Wed, Jul 08, 2026, 11:18 PM UTC

Dylan16807 wrote: post #228

bensyverson wrote: post #201

Earlier quoted context omitted.

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

> The odds of this happening go way up the larger your individual drives are. This is a common claim, but honestly, citation needed. We can't just apply bit error rates from the datasheet that haven't been updated in 15 years. I'm sure a 2-day rebuild is a little more risky than a 4-hour rebuild, but I'm not convinced it's by all that much. Especially if you had a monthly scrub going to prevent disk rot and disks sec…

>> The odds of this happening go way up the larger your individual drives are.

> This is a common claim, but honestly, citation needed.

FWIW, here's the quick & dirty argument for that:

Assume:

- We're working within a given generation of disk technology, with comparable drive and interface tech.

- The drives' I/O speed is relatively constant, not scaling with the drive capacity.

- The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.

- The drives, regardless of size, will be some fixed-ish percentage full. (If nothing else, the boss won't sign off on overspending for unneeded capacity.)

From there - the time needed for resilvering scales linearly with the drive size. As does the number of non-redundant bits that could go bad in any given second of the resilvering. Linear time linear - so the chance of data loss during resilvering rises with the square of the disk size.