I had written a Wifi based LTSP for Linux sometime ago.
http://www.sarathlakshman.com/2010/03/14/wireless-ltsp/
CoreOS: Boot on Bare Metal with PXE
41–50 of 58 posts
Re: CoreOS: Boot on Bare Metal with PXE
#42Earlier quoted context omitted.
I've been looking at a mSATA SSD and a low profile pcie card for our Ceph cluster to free up the OS/journal drive slots for more 4tb spinning disks.
How are you finding ceph?
We've been using it for about 15 months now without any problems with RADOS. Last winter we had some data loss with CephFS and needed to rebuild the filesystem from backup, but CephFS is unsupported so it was somewhat expected. I think the issues came from the Linux kernel client (circa the 3.2 kernel iirc), so we switched over to the FUSE client instead and have been on that since.
Good news is that performance is much, much better with 0.61 than prior releases. Both for RADOS and CephFS. We'll probably upgrade to 0.67 in the next few weeks and it's probably time to upgrade to the 3.10 kernel as well for btrfs fixes and to kick the tires on the kernel fs client again.
Re: CoreOS: Boot on Bare Metal with PXE
#43For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network. A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and s…
"Back in the day" (pre-PXE), we had rooms full of machines -- without hard drives -- that booted completely across the network and mounted all of their filesystems via NFS. https://en.wikipedia.org/wiki/Diskless_node Over the last few years, I've wondered why server vendors don't ship servers with type of flash-based storage (or similar) -- perhaps 4-8 GB -- that's large enough to hold an installation of (for example…
VMware has a product to do exactly this (Auto Deploy). It PXE boots ESXi and can configure it automatically.
Re: CoreOS: Boot on Bare Metal with PXE
#44Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
If you are ok with CentOS/RHEL, there is a scalable way to boot multiple machines simulataneously using ........ bittorrent !
http://www.rocksclusters.org/rocks-doc/papers/two-pager/pape...
Rocks is used at production cluster installations with thousands of servers.
http://www.rocksclusters.org/rocks-register/index.php?sortby...
Re: CoreOS: Boot on Bare Metal with PXE
#45I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure.
>I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure.
I find comments like these amusing. Sysadmins have been using PXE to boot servers... for quite some time now. Long before joyent existed, and long before VMware was something you'd seriously run a server under. (vmware came about around the time of the 2.1 version of the PXE standard, which was when I was first getting my feet wet; I didn't seriously start using pxe until the early oughts.)
Hell, /I/ built a nfs/pxe diskless cluster before joyent existed. As part of that, I demo'd a 'initrd only' system like this coreos thing, only we were using FreeBSD. We ended up going with / on nfs; it was way easier to update.
That said, I'm not knocking CoreOS; I might even use this. Maintaining your own bootable initrd with root filesystem is work. I currently use distro 'rescue images' (for centos, at least, you append 'rescue' to the installer, and it downloads a small initrd /... but it's less than optimal.)
I mean, I'm not shitting on joyent, either; I think most of the value they bring is managing this shit ongoing, which is not a trivial amount of work. I mean, the whole idea behind companies like Joyent is to make is so you don't need a me screwing with your dhcp server, and there's value in that. I'm grouchy and charge a lot of money.
Re: CoreOS: Boot on Bare Metal with PXE
#46Earlier quoted context omitted.
> I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. Heh, we were doing this 20-25 years ago. It's funny sometimes how I.T. seems to circle back around.
>> stateless immutable servers [emphasis added] >Heh, we were doing this 20-25 years ago. It's possible, but I'd be surprised (unless you mean X servers in the crazy X-windows server-is-really-what-any-sane-person-would-call-a-client sense). Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk.
maybe not 25 years ago... but it was common in the early '00s. This is not new tech. I participated in a project to do this with FreeBSD and nfs, I think around '03 or so. It was a smallish cluster (under 100 nodes) built to handle a simple Freebsd Apache Mysql PHP stack for a growing realestate ASP. Even then, it was not a new thing; there was plenty of documentation from other folks doing it.
I mean, it didn't go as smoothly as pxe booting does today; we ended up needing to reflash all our network cards to make it work, something I have not had to do in the last 5 years.
Re: CoreOS: Boot on Bare Metal with PXE
#47Earlier quoted context omitted.
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo. Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it.
The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.
Re: CoreOS: Boot on Bare Metal with PXE
#48Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a net…
We commonly reboot entire clusters at once (around 10,000 servers in larger clusters -- each running a full Linux OS) over PXE without a problem. We have a configuration management machine that creates an image, then we push that down to a small cluster of TFTP servers that serve it out. The strain on NFS (we keep parts of the OS in RAM, and load other parts on demand over NFS) after we kexec from the PXE kernel into…
Since you mention kexec and TFTP+NFS, are you currently using Perceus? Or is there another system out there with that combo?
Re: CoreOS: Boot on Bare Metal with PXE
#49Earlier quoted context omitted.
>> stateless immutable servers [emphasis added] >Heh, we were doing this 20-25 years ago. It's possible, but I'd be surprised (unless you mean X servers in the crazy X-windows server-is-really-what-any-sane-person-would-call-a-client sense). Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk.
The NCD X terminals I used to work on did this (they actually ran a BSD Unix kernel under the hood, I believe). Also was popular for some Sun diskless workstations -- although the protocol was bootp instead of pxe.
> although the protocol was bootp instead of pxe
Well, they used bootp followed by tftp, and so did most other UNIX workstations of that time period. DHCP is effectively a superset of bootp -- it uses the same UDP port. PXE is DHCP+tftp along with some specification of tftp paths, where the image is loaded in RAM, etc.
Really the overall netboot mechanism is nearly identical as it was 25 years ago, just some protocol tweaks and a new acronym.
Re: CoreOS: Boot on Bare Metal with PXE
#50Since I asked for this in the original CoreOS thread, let me be the first to say thanks. I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. I recommend experimenting with PXE in Vagrant or o…
What are the upsides/downsides of this? It's such a paradigm shift, I'm having a hard time thinking of any advantages.