CoreOS: Boot on Bare Metal with PXE
coreos.com
CoreOS: Boot on Bare Metal with PXE
1–10 of 58 posts
Re: CoreOS: Boot on Bare Metal with PXE
#2Since I asked for this in the original CoreOS thread, let me be the first to say thanks. I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on.
BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. I recommend experimenting with PXE in Vagrant or on a separate physical network to avoid breaking your production DHCP.
Re: CoreOS: Boot on Bare Metal with PXE
#3Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF.
Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).
Re: CoreOS: Boot on Bare Metal with PXE
#4Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
Re: CoreOS: Boot on Bare Metal with PXE
#5Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.
Re: CoreOS: Boot on Bare Metal with PXE
#6Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo.
Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
Re: CoreOS: Boot on Bare Metal with PXE
#7So is this basically LFS with built-in PXE boot support and orientation towards servers?
Re: CoreOS: Boot on Bare Metal with PXE
#8Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
So, a 'PXE server' is nothing more then a DHCP server and some method of fetching files (historically, TFTP, but this is not a requirement).
ISC DHCPD is pretty bulletproof. Many large carriers/ISPs use it, so scaling this end should be pretty straightforward. You'd probably want to statically configure leases for all your servers though.
Pretty much all PXE roms built into NICs only support TFTP. I would use TFTP to load a iPXE, then use iPXE to load the operating system over something else (probably HTTP). Scaling HTTP is well understood (this is about the easiest HTTP scaling you can do, downloading a few static files).
Nothing about this is hard to scale, nor does it require any special network setup.
Re: CoreOS: Boot on Bare Metal with PXE
#9Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
We commonly reboot entire clusters at once (around 10,000 servers in larger clusters -- each running a full Linux OS) over PXE without a problem. We have a configuration management machine that creates an image, then we push that down to a small cluster of TFTP servers that serve it out. The strain on NFS (we keep parts of the OS in RAM, and load other parts on demand over NFS) after we kexec from the PXE kernel into the production kernel causes more problems than the initial TFTP traffic (but it usually works fine as well). Btw, after booting, we use PanFS (DirectFlow) or Lustre for computing stuff, not NFS.
Although it's not what we use, here's a program that does a similar type of management: http://warewulf.lbl.gov/trac If you take the time to combine Warewulf with something like Puppet or Chef, you'll have a nice system for managing 100s of thousands of machines (I could easily see this scaling to over a million servers if you have the cash to build something like that).
If you're wondering about dynamic libraries in an environment like this, take a look at https://github.com/hpc/Spindle
And yes, I still get giddy when I type one command to reboot 10,000 servers.
Re: CoreOS: Boot on Bare Metal with PXE
#10For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network.
A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and servers), can make a DHCP request (outside of an operating system), download via TFTP, then boot a kernel/initial ram disk, say for example an OS installer, a LIVE CD, or CoreOS.
+-----> #1 PXE makes DHCP request,
| redirects to TFTP server,
| loads kernel/initial ram disk via TFTP
|
| +--+ #2 PXE boots kernel/initial ram disk
| |
+ v
+------------------+
| PXE Network Card |
|------------------|
| server |
| hardware/os |
+------------------+
In summary, it looks like CoreOS provides you with their kernel and initial ram disk [2], then you can just boot the machine over the network without actually installing anything. Basically running everything out of RAM, just like a LIVE CD, most likely with the option to install to disk for persistent storage of your images, etc.
ps. It is common to use NFS to mount persistent storage in these environments too, or if you are using a HPC environment, then Luster or something like that.
[1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment