Live data from Hacker News

Building a Budget Homelab NAS Server

mtlynch.io

201–210 of 316 posts

Re: Building a Budget Homelab NAS Server

#201
I would recommend anyone building a home NAS like this in 2022 to look into buying some slightly older 10GbE network interfaces on ebay (an intel X520-DA2 with 2 x 10Gbps SFP+ ports can be found for $55) as a PCI-E card. It's not hard to exceed the transfer ability of an ordinary 1000BaseT port to a home switch these days.

And if you have just a few powerful workstation desktop PCs it's also worth it to connect them at 10GbE to a new switch.

here's a fairly typical one. these have excellent freebsd and linux kernel driver support.

https://www.ebay.com/itm/265713815725?epid=1537630441&hash=i...

Re: Building a Budget Homelab NAS Server

#202

Earlier quoted context omitted.

I also forgot to mention, your guidance on ZFS memory requirements is outdated. From what I have heard recent releases have drastically reduced the size of ARC cache necessary. One person reported it working phenomenally on a newer Raspberry Pi.

True but COW is even harder on an SD than ext4 is so I would really not use it on a pi unless it's not using SD storage :)

Been using ZFS on eMMC with things like Orange Pi-s and Rock64-s for a few years, so far works good for me.

Re: Building a Budget Homelab NAS Server

#203

Earlier quoted context omitted.

> any read error on any of the remaining disks is a lost/corrupted file. That is the meat of it. With traditional RAID it is the same issue, except you never know it happens because as long as the controller reads something, it's happy to replicate that corruption to the other disks. At least with ZFS, you know exactly what was corrupted and can fix it, with traditional RAID you won't know it happened at all until yo…

You don't need to fix anything. ZFS automatically self-heals an inconsistent array (for example if one mirrored drive does not agree with the other, or if a parity drive disagrees with the data stripe.) ZFS does not suffer data loss if you "suffer a total disk failure." I have no idea where you're getting any of this from.

If the data on disk (with no redundant copies) is bad, you’ve (usually) lost data with ZFS. It isn’t ZFS’s fault, it’s the nature of the game.

The poster built a (non redundant) zfs pool on top of a hardware raid6 device. The underlying hardware device had some failed drives, and when rebuilt, some of the underlying data was lost.

ZFS helped by detecting it instead of letting the bad data though like would normally have happened.

Re: Building a Budget Homelab NAS Server

#204
I have tried a few different OSes but my favorite by far is unRaid. It’s really easy to setup and maintain and it gave me a lot of really good experience with server maintenance and the whole container ecosystem. I bought a 24 drive server chassis and am slowly filling it up. Up to 80 TB now and I only have to have one extra drive for local backup (I also backup to another box that I do periodically).

Re: Building a Budget Homelab NAS Server

#205
Good article. He choose a fractal design case which I really like (the company not the specific model).

I had all kinds of thermal problems with a too small case that I used for my truenas build. It would turn off without any trace in server logs (I have real server HW and therefore expected something in logs since there is a whole separate computer for this).

I changed the case from a NAS case to another fractal desfrign case with lots of space for drives and heatsink. All thermal issues disappeared.

I just wanted to warn anyone who is building to take this seriously. Some HW drives generate a lot of heat.

Re: Building a Budget Homelab NAS Server

#206

Earlier quoted context omitted.

ZFS on Root just sounds like pain to me. I opt for MD RAID on root and then ZFS my other volumes. I would also say Ubuntu is probably the better choice for Linux ZFS, as CentOS seems to be lacking good support.

Once you try it, you're never going back. Snapshots are made for things like system administration. Upgrade borked your system? Just rollback. Want to use the last version of your firewall config? I wrote a utility you might like to try, httm[1], which allows you to restore from your snapshot-ed unique versions. If you like ZFS, then trust me you have to have ZFS on root. [1]: https://crates.io/crates/httm

Had you previously done a Show HN on this? I feel like I saw it once before.

Re: Building a Budget Homelab NAS Server

#207
post #49

Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server. Any questions or feedback about the post are more than welcome.

I have a couple of home built TrueNAS systems for many years (since FreeNAS, ~ 10 years ago), here is some feedback:

- with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster.

- you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed

- unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else *

- * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it.

- that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports

- TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA

- your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%.

- that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU.

- GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it.

- unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network

Re: Building a Budget Homelab NAS Server

#208
I chose the same case for my NAS. Main thing I did different was rather than buying a consumer board, I bought a mini-ITX Xeon-D board from supermicro which had integrated dual 10G NICs, 6x SATA, and an ASPEED IPMI for remote management. Was $400 for that board a few years ago (soldered CPU).

Re: Building a Budget Homelab NAS Server

#209
post #49

Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server. Any questions or feedback about the post are more than welcome.

- "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does, but cheaper. ZFS snapshots obviously don't cover other failure modes like natural disasters or a break in, so RAID is still not a backup. - for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This…

> - "RAID is not a backup" primarily because "you could rm -rf".

or your house could burn down

or somebody could steal the computer while you're away on vacation

or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage

or your house could flood

lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.

Re: Building a Budget Homelab NAS Server

#210
I am sure the author will appreciate ditching the proprietary Synology to go instead with a custom ZFS server, as the reliability, recoverability, and feature set of ZFS is quite frankly hard to beat. I have been using ZFS to build my custom NASs for the last... checks notes 17 years. I started back when ZFS was only available on Solaris/OpenSolaris. My builds usually have between 5 and 7 drives (raidz2).

However I do not recommend his choice of 4 x 8TB drives in a raidz1. Financially and technically it doesn't make sense. He spent $733 for 24TB usable ($30.5/TB)

He should have bought fewer, larger drives. For example 14TB drives sell for $240. So a config with 3 x 14TB in a raidz1 would total $720 for 28TB usable ($25.7/TB). Smaller costs, more storage, one less drive (= increased reliability)! It's win-win-win.

Especially given his goal and hope is in a couple years to be able to add an extra drive and reshape the raidz1 to gain usable space, then a 14TB drive then will be significantly cheaper per TB than an 8TB drive (today they are about the same cost per TB).

Actually, with only 8.5TB of data to store presently, if I were him I would probably go one step further and go with a simple zfs mirror of 2 x 18TB drives. At $320 per drive that's only $640 total for 18TB usable ($35.6/TB). It's a slightly higher cost per TB (+17%), but the reliability is much improved as we have only 2 drives instead of 4, so totally worth it in my eyes. And bonus: in a few years he can swap them out with 2 bigger-capacity drives, and ZFS already supports resizing mirrors.

Post reply on HN