Diskless Linux boot using ZFS, iSCSI and PXE

Pretty cool! You could also boot into an ephemeral minimal initrd that displays a selection menu instead of doing it in iPXE. That would grab the new kernel and initrd from the network and kexecs it without reboot.

Re: Diskless Linux boot using ZFS, iSCSI and PXE

#7

Post by yjftsjthsd-h » Thu, May 07, 2026, 5:34 AM UTC

Nice. I'm extra fond of ZFS backed network root filesystem, because it lets you put an OS on ZFS without needing to deal with ZFS support in that OS. (One of these days I want to try OpenBSD with its root on NFS on ZFS, either from Linux or FreeBSD.)

Does anyone have an opinion on iSCSI vs NBD?

Re: Diskless Linux boot using ZFS, iSCSI and PXE

#8

Post by ahepp » Thu, May 07, 2026, 5:34 AM UTC

You might find it worth upgrading to 10gbps if you continue to go down this road. The Mikrotik CRS-309 has served me well, and a couple Intel X520-DA2s. I believe those NICs can do iSCSI natively, and pass the session to the operating system with iBFT.

SFP28 might be cheap enough now too, I'm not sure...

Re: Diskless Linux boot using ZFS, iSCSI and PXE

#9

Post by dhash » Thu, May 07, 2026, 5:39 AM UTC

something worth mentioning here is that iSCSI is quite unhappy on congested networks or packet loss caused by incast traffic.

to make this actually work well, consider modifying your switches QoS settings to carve out a priority VLAN for iSCSI traffic

Re: Diskless Linux boot using ZFS, iSCSI and PXE

#10

Post by contingencies » Thu, May 07, 2026, 5:40 AM UTC

ahepp wrote: post #5

ggm wrote: post #3

NFS diskless is the more common approach I've used but this is very cool.

When I tried root-on-nfs I had a lot of issues. The Redhat and Arch package managers don't seem to like it (presumably a sqlite thing?).

You can download the rootfs, extract it to a ramdisk, and just run in memory. This is fast for everything. Unfortunately, memory just got super expensive. Fortunately, Linux requires ~no memory to do many useful things.