Running CoreOS on Intel NUC's via PXE boot from a HP Microserver; a poor man's blade chassis/datacenter?
CoreOS: Boot on Bare Metal with PXE
31–40 of 58 posts
Re: CoreOS: Boot on Bare Metal with PXE
#32Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
So, a 'PXE server' is nothing more then a DHCP server and some method of fetching files (historically, TFTP, but this is not a requirement). ISC DHCPD is pretty bulletproof. Many large carriers/ISPs use it, so scaling this end should be pretty straightforward. You'd probably want to statically configure leases for all your servers though. Pretty much all PXE roms built into NICs only support TFTP. I would use TFTP to…
iPXE is awesome.
My standard install is to embed a script into iPXE that chainloads a config file over TFTP (this works properly with non-ISC DHCPD servers, like OpenBSD's fork and dnsmasq), then loads various OS's and diagnostic tools over HTTP or NFS.
So much easier than slinging around optical media or hard disks for installation/diagnostic/booting diskless clients.
Re: CoreOS: Boot on Bare Metal with PXE
#33Earlier quoted context omitted.
"Back in the day" (pre-PXE), we had rooms full of machines -- without hard drives -- that booted completely across the network and mounted all of their filesystems via NFS. https://en.wikipedia.org/wiki/Diskless_node Over the last few years, I've wondered why server vendors don't ship servers with type of flash-based storage (or similar) -- perhaps 4-8 GB -- that's large enough to hold an installation of (for example…
Well, there's the solution of putting a USB flash drive inside the server's case (optionally securing it with duct tape). In fact, a USB flash drive is the recommend medium for booting FreeNAS.
I've seen recent servers also ship with an SD or MicroSD slot on the motherboard too. Probably a bit more secure than a USB stick that can come unplugged easily.
Re: CoreOS: Boot on Bare Metal with PXE
#34Earlier quoted context omitted.
> I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. Heh, we were doing this 20-25 years ago. It's funny sometimes how I.T. seems to circle back around.
>> stateless immutable servers [emphasis added] >Heh, we were doing this 20-25 years ago. It's possible, but I'd be surprised (unless you mean X servers in the crazy X-windows server-is-really-what-any-sane-person-would-call-a-client sense). Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk.
The NCD X terminals I used to work on did this (they actually ran a BSD Unix kernel under the hood, I believe). Also was popular for some Sun diskless workstations -- although the protocol was bootp instead of pxe.
Re: CoreOS: Boot on Bare Metal with PXE
#35For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network. A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and s…
"Back in the day" (pre-PXE), we had rooms full of machines -- without hard drives -- that booted completely across the network and mounted all of their filesystems via NFS. https://en.wikipedia.org/wiki/Diskless_node Over the last few years, I've wondered why server vendors don't ship servers with type of flash-based storage (or similar) -- perhaps 4-8 GB -- that's large enough to hold an installation of (for example…
Server vendors have shipped that for quite a while now; Dell for example has offered an option on servers where ESXi is pre-installed on an SD or CF card built into the system for at least five years.
Re: CoreOS: Boot on Bare Metal with PXE
#36For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network. A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and s…
"Back in the day" (pre-PXE), we had rooms full of machines -- without hard drives -- that booted completely across the network and mounted all of their filesystems via NFS. https://en.wikipedia.org/wiki/Diskless_node Over the last few years, I've wondered why server vendors don't ship servers with type of flash-based storage (or similar) -- perhaps 4-8 GB -- that's large enough to hold an installation of (for example…
Many of them do. Many newer HP blades, for example, come with an SD card inside to do exactly that with either local or network storage.
Re: CoreOS: Boot on Bare Metal with PXE
#37Earlier quoted context omitted.
> BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. You don't even need a config file. Here's a command line snippet I have saved for testing PXE installs: $ sudo dnsmasq -hdq -i en0 -p0 --enable-tftp --tftp-root=`pwd` -Mpxelinux.0 -F10.10.10.100,10.10.10.199
So this snippet makes the machine you run it from into the PXE Server?
Yes.
Re: CoreOS: Boot on Bare Metal with PXE
#38Earlier quoted context omitted.
It is possible to PXE boot a full blown OS, not just OS installer, Live CD or CoreOS. The product usually used to do this is: http://support.citrix.com/proddocs/topic/technologies/pvs-pr... It is just a nice GUI that ties into PXE and TFTP and just streams an image from a disk file. On a 1Gb network I have booted over 300 physical machines in under 2 minutes simultaneously. For those curious, unfortunately PVS is pac…
The product usually used to do this is Linux...
Yea I meant "I usually" :-p
Re: CoreOS: Boot on Bare Metal with PXE
#39Earlier quoted context omitted.
"Back in the day" (pre-PXE), we had rooms full of machines -- without hard drives -- that booted completely across the network and mounted all of their filesystems via NFS. https://en.wikipedia.org/wiki/Diskless_node Over the last few years, I've wondered why server vendors don't ship servers with type of flash-based storage (or similar) -- perhaps 4-8 GB -- that's large enough to hold an installation of (for example…
I've been looking at a mSATA SSD and a low profile pcie card for our Ceph cluster to free up the OS/journal drive slots for more 4tb spinning disks.
How are you finding ceph?
Re: CoreOS: Boot on Bare Metal with PXE
#40Earlier quoted context omitted.
It is possible to PXE boot a full blown OS, not just OS installer, Live CD or CoreOS. The product usually used to do this is: http://support.citrix.com/proddocs/topic/technologies/pvs-pr... It is just a nice GUI that ties into PXE and TFTP and just streams an image from a disk file. On a 1Gb network I have booted over 300 physical machines in under 2 minutes simultaneously. For those curious, unfortunately PVS is pac…
DRBL can do this for Linux (if you require some software to set it up for you). You can actually accomplish this with nothing more then the linux kernel if you're clever: https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs...
Thanks I have not heard of DRBL. After reading it sounds pretty much like PVS, though PVS does have some additional features I don't see on it. Will have to try it out! DRBL is a free software though.