Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub's Metal Cloud

githubengineering.com

Post reply on HN
Print view View original

11–20 of 83 posts

Re: GitHub's Metal Cloud

#11

Post by mverwijs »

We did something similar at Optiver.

    * boot a custom live cd (a la Knoppix) over PXE
    * Live CD places node into database if it doesn't exist yet, 
    using dmidecode to find serialnumbers and such
    * Live Cd keeps querying database for instructions
    * Engineer adds a profile to the node in the database
    * Live Cd slices up disk to match the profile
    * Live Cd fetches a tarball of the base OS from an URL 
    and throws it on the metal. Runs grub setup. Reboots.
Tumblr did something like this with http://tumblr.github.io/collins/.

Re: GitHub's Metal Cloud

#12

Post by mverwijs »

sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE.

   * boot a live image into memory
   * point LXD/RKT/Docker to /containers
   * ...
   * profit!

Re: GitHub's Metal Cloud

#13

Post by shanselman »

alpb wrote: post #4

> We've hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we've hit a failure or not. That's pretty funny yet sounds a lot familiar to many of us as every now and then we all do these sort of nasty hacks.

Ya, I came here to talk about that line as well. The whole article was interesting, but that line stood out as a pretty surprising hack for 2015.

Re: GitHub's Metal Cloud

#14

Post by iofj »

mverwijs wrote: post #12
sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!

The big issue I'm having with that is that it involves trusting vendors to get network boot right. Especially when it comes to the looping part of "loop until DHCP gets a response" it becomes a problem. One of the cheap vendor tries 30 times and then goes to a boot failed screen after trying the disk.

Also, 1 time out of a 4-5000 or so network boot fails. Not sure why.

Re: GitHub's Metal Cloud

#16

Post by icebraining »

lwhalen wrote: post #9
sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all , and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and s…

Containers are just a namespacing tool, though; you're still running on bare metal (well, bare Linux). Docker in particular runs on AuFS, which is slow, but other containerization tools just use a chroot.

Re: GitHub's Metal Cloud

#18

Post by bcantrill »

mverwijs wrote: post #12
sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!

For whatever it's worth, SmartDataCenter, Joyent's open source SmartOS-based system for operating a cloud[1], does exactly this[2] -- and (as explained in an old but still technically valid video[3]) we have found it to be a huge, huge win. And we even made good on the Docker piece when we used SDC as the engine behind Triton[4] -- and have found it all to be every bit as potent as you suggest![5]

[1] https://github.com/joyent/sdc

[2] https://github.com/joyent/sdc/blob/master/docs/developer-gui...

[3] https://www.youtube.com/watch?v=ieGWbo94geE

[4] https://www.joyent.com/blog/triton-docker-and-the-best-of-al...

[5] https://www.joyent.com/blog/spin-up-a-docker-dev-test-enviro...

Re: GitHub's Metal Cloud

#20

Post by stephenr »

These lines seem odd to me, maybe it's just the wording:

> [gPanel] Deploying DNS via Heaven... > hubot is deploying dns/master (deadbeef) to production. > hubot's production deployment of dns/master (deadbeef) is done! (6s)

Is this just an IMO odd use of the word "deploying" or does a DNS change really mean building and deploying a new package/image?

Post reply on HN