If your budget server starts with a first step of buying new hardware, I'm going to ignore your advice. A 500W psu? nope. Buy a used thinkcentre with a 250W CPU for $50-$100 and save the planet more of this e-waste.
Building a Budget Homelab NAS Server
221–230 of 316 posts
Re: Building a Budget Homelab NAS Server
#222Earlier quoted context omitted.
I bought brand new 10 Gbps NICs ($47/pcs) and switch ($120) for less than that and replacements are readily available. The 20m AOC cables were indeed $40. A NAS home build of this size will never exceed 10 Gbps, I barely get ~ 2 Gbps out of the spinning disks.
the main application where one will see real 10Gbps speeds is if the NAS also has a fast NVME SSD used for smaller file size needs at high speeds... for instance I have a setup which is meant for working with uncompressed raw yuv420 or yuv422p 1080p and 4K video, there's a 512GB NVME SSD and a 1TB SSD set up as individual JBOD and exposed to the network for video editing scratch file storage, and it will definitely s…
Very valid point, but I don't know how fast you will hit the chipset speed limits; that NIC is connected to the chipset that is connected to the CPU via a 4x PCIe link and from there to the nVME with 4x PCIe link. In theory you have 4 or 8 GB/sec max bandwidth, but the CPU-chipset link is not dedicated. If you go for Threadripper the math is very different, you have lots of direct CPU connections for (multiple) NIC and multiple nVME.
Re: Building a Budget Homelab NAS Server
#223I am sure the author will appreciate ditching the proprietary Synology to go instead with a custom ZFS server, as the reliability, recoverability, and feature set of ZFS is quite frankly hard to beat. I have been using ZFS to build my custom NASs for the last... checks notes 17 years. I started back when ZFS was only available on Solaris/OpenSolaris. My builds usually have between 5 and 7 drives (raidz2). However I d…
> For example 14TB drives sell for $240 Where? Also, is it worthwhile to buy hard drives explicitly for NAS when you're using ZFS? For example, Seagate has the IronWolf product line explicitly for NAS and cost more.
Re: Building a Budget Homelab NAS Server
#224Nice build. I recently built my second NAS (from a used R720 from ebay). The total (without disks) is pretty similar to the build documented in this article. Having a large NAS has had an interesting (though predictable) impact on all the computers around it: Pretty much every bit of data lives on the NAS (accessed either by CIFS, NFS, or iSCSI). When I had to reinstall Windows, it was mostly painless because all my…
How is performance, especially with regards to load times, if your steam library is mounted remotely? I ask because the difference between a SSD and a hard drive can be massive in this regards, so I'd be really interested to know if the network latency is also a comparable hit.
The performance is fine. It's been years since I ran my steam library on HDD, so I don't have anything really to compare to (except my own expectations and impatience). The NAS is running 7 SSD drives in zfs raid, and exports a 1TB volume over iSCSI, via a dedicated 10GBe fiber link. Anecdotally, I will often boot into games faster than friends who have a local SSD (so I think this means that I've gotten disk perf to be "good enough" that other hardware components start to dominate things like load times)
Re: Building a Budget Homelab NAS Server
#225Earlier quoted context omitted.
the main application where one will see real 10Gbps speeds is if the NAS also has a fast NVME SSD used for smaller file size needs at high speeds... for instance I have a setup which is meant for working with uncompressed raw yuv420 or yuv422p 1080p and 4K video, there's a 512GB NVME SSD and a 1TB SSD set up as individual JBOD and exposed to the network for video editing scratch file storage, and it will definitely s…
Very valid point, but I don't know how fast you will hit the chipset speed limits; that NIC is connected to the chipset that is connected to the CPU via a 4x PCIe link and from there to the nVME with 4x PCIe link. In theory you have 4 or 8 GB/sec max bandwidth, but the CPU-chipset link is not dedicated. If you go for Threadripper the math is very different, you have lots of direct CPU connections for (multiple) NIC a…
you may be referencing the original post's cpu/motherboard combo which is not the same as the pci-e bus/lanes, slot and setup on my home file server and desktop PC.
Re: Building a Budget Homelab NAS Server
#226I would recommend anyone building a home NAS like this in 2022 to look into buying some slightly older 10GbE network interfaces on ebay (an intel X520-DA2 with 2 x 10Gbps SFP+ ports can be found for $55) as a PCI-E card. It's not hard to exceed the transfer ability of an ordinary 1000BaseT port to a home switch these days. And if you have just a few powerful workstation desktop PCs it's also worth it to connect them…
Or go a little further and spend ~$200 on a used passive FDR Infiniband switch, $50 per used dual port FDR IB NIC, $40 for a 10 meter fibre optic cable including transceivers at each end. Then run IP over IB on each host and you have a 56 Gbit network that all your applications will just see as another network interface on each host.
Can you share what your experience with implementing IPoIB with used gear was? I'm asking mainly because I actually got interested recently with such setups however I got rather discouraged by the driver support.
As an example here is the driver page for Mellanox, now owned by Nvidia, since they are a major Infiniband equipment supplier: https://network.nvidia.com/products/infiniband-drivers/linux...
It seems that some decent support only exists for more recent generations. The older ones like ConnectX-3 or earlier, which typically show up on ebay are either not supported any more or maybe available for older kernel versions and soon to be EOLed.
So do I understand it correctly that to use such adapters one has to actually downgrade to an older kernel version?
Or is there some basic support in the latest Linux kernels for older generations still?
Re: Building a Budget Homelab NAS Server
#227I am sure the author will appreciate ditching the proprietary Synology to go instead with a custom ZFS server, as the reliability, recoverability, and feature set of ZFS is quite frankly hard to beat. I have been using ZFS to build my custom NASs for the last... checks notes 17 years. I started back when ZFS was only available on Solaris/OpenSolaris. My builds usually have between 5 and 7 drives (raidz2). However I d…
> For example 14TB drives sell for $240 Where? Also, is it worthwhile to buy hard drives explicitly for NAS when you're using ZFS? For example, Seagate has the IronWolf product line explicitly for NAS and cost more.
I looked at a popular line (Seagate X16) here:
https://www.amazon.com/dp/B07T63FDJQ - $237.99 right now
Drives branded for NAS applications differ slightly from mainstream drives. For example Seagate claims the IronWolf is "designed to reduce vibration, accelerate error recovery and control power consumption" which essentially means the drive head actuators will be operated more gently (reduced vibration) which slightly increases latency and slightly reduces power consumption, and also the firmware is configured so that it does fewer retries on I/O errors, so the disk commands time out more quickly in order to pass the error more quickly to the RAID/ZFS layer (why wait a minute of hardware retries when the RAID can just rebuild the sector from parity or mirror disks.) IMHO for home use, none of this is important. Vibration is only an issue in flimsy chassis, or extreme situations like dozens of disks packed tightly together, or extreme noise as found in a dense data center (see the video of a Sun employee shouting at a server). And whether you have to wait a few seconds vs a few minutes for an I/O operation to timeout when a disk starts failing is completely unimportant in a non-business critical environment like a a home NAS.
Re: Building a Budget Homelab NAS Server
#228Earlier quoted context omitted.
Once you try it, you're never going back. Snapshots are made for things like system administration. Upgrade borked your system? Just rollback. Want to use the last version of your firewall config? I wrote a utility you might like to try, httm[1], which allows you to restore from your snapshot-ed unique versions. If you like ZFS, then trust me you have to have ZFS on root. [1]: https://crates.io/crates/httm
Had you previously done a Show HN on this? I feel like I saw it once before.
Someone else posted about it awhile ago: https://news.ycombinator.com/item?id=31184404
Re: Building a Budget Homelab NAS Server
#229If you're going for budget, decommissioned data center rack servers work well.
I purchased a dual xeon (24 cores total) with 64gb of memory, 12 3.5" bays, dual power supplies, for about $250 from a liquidator.
Filling it with HDDs was pricey but you can expand as you need it to spread the expenditure.
Re: Building a Budget Homelab NAS Server
#230Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server. Any questions or feedback about the post are more than welcome.
- "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does, but cheaper. ZFS snapshots obviously don't cover other failure modes like natural disasters or a break in, so RAID is still not a backup. - for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This…
> - "RAID is not a backup" primarily because "you could rm -rf". ZFS snapshots cover that failure mode to the same extent that synchronization with offsite does
Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally.
(Also fire, compromise, police confiscation, etc.)