Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub's Metal Cloud

githubengineering.com

Post reply on HN
Print view View original

41–50 of 83 posts

Re: GitHub's Metal Cloud

#41

Post by mpdehaan2 »

mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

I did something sort of similar with Live CD images for booting hardware not on a PXE network in Cobbler. It could also register machines seen then first time by making system records for them.

https://github.com/cobbler/cobbler

What is interesting in this article in particular are the auto-firmware upgrade state transitions, which seem pretty neat.

The chat bot is I guess neat, but if a lot of people are using the cloud that could be a hard way to get status.

Re: GitHub's Metal Cloud

#42

Post by mverwijs »

voltagex_ wrote: post #37
mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Hey cool so what I'm fiddling around with in iPXE isn't completely obsolete in the age of Docker and disposable VMs.

Well, no. But IMHO, iron is only do-able if you have enough of it. This I learned the hard way at my previous contract, where we only had a handful of servers and all of them production.

When you want to automate your complete infra, including rolling out hardware, you need hardware to develop and test on. Entropy of life will ensure that exactly that moment you need to reinstall that PostgreSQL slave from scratch, the PXE server is unreachable, or the server has a different diskcontroller, or an iLO certificate expired. Or something stupid.

Test your code. And for Ops this means: machines that are solely for the testing pleasure of Ops. No other function.

Re: GitHub's Metal Cloud

#43

Post by invertedohm »

mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Kind of amazing that the state of the art in this area hasn't changed in nearly 10 years. The marketing angle is funny as well - everything has to be a cloud now. I think I used to call what we built at Optiver a "private cloud" but "Metal Cloud" is nicely buzzwordy as well.

PS - Hi Marty, unknown nick here but I'm sure you can figure out who I am. :D

Re: GitHub's Metal Cloud

#45

Post by jsjohnst »

mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Actually there's heavy overlap in functionality if you include other Tumblr things with Collins like Phil, Genesis, and Configrr.

Collins - asset management

Phil - ipxe booting based on Collins state

Genesis - base hardware configuration (firmware, bios, raid, etc), burnin, kickstart

Configrr - state / config management

Re: GitHub's Metal Cloud

#46

Post by vidarh »

voltagex_ wrote: post #37
mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Hey cool so what I'm fiddling around with in iPXE isn't completely obsolete in the age of Docker and disposable VMs.

Even those machines running Docker and disposable VMs still need to run on something.

I've got systems PXE booting to CoreOS where everything else runs in Docker containers (even the odd KVM VM).

Re: GitHub's Metal Cloud

#47

Post by osxrand »

alpb wrote: post #4

> We've hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we've hit a failure or not. That's pretty funny yet sounds a lot familiar to many of us as every now and then we all do these sort of nasty hacks.

Hah! I wrote a script a few months back and had to solve an issue of figuring out the state of a program running under wine, and that was my solution (to not in ruby, just a quick bash script). I was pretty happy with the results but if felt like an incredibly crude way to solve the problem, now I read that this is being used at much higher levels than I'll get to, maybe it's not so bad :)

Re: GitHub's Metal Cloud

#48

Post by vidarh »

mverwijs wrote: post #12
sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!

CoreOS lets you do this if you PXE boot. It will then by default run entirely out of RAM.

Re: GitHub's Metal Cloud

#50

Post by wtbob »

ptype wrote: post #49

Why would a company like GitHub choose Ubuntu over Debian? The LTS policy?

I too would be interested in the answer. From my perspective, Debian is the server Linux distro par excellence, and in my experience the folks who choose Ubuntu have been devs who don't actually use Linux (e.g., the sorts who develop on a Mac or in a VM rather than on a personal Linux system). It's not really fair to Ubuntu, which is decent enough in its own way, but I tend to consider the choice of Ubuntu to be a bit of an architecture smell.

I'm honestly interested in what the valid reasons to prefer Ubuntu over Debian (particularly on the server side) are.

Post reply on HN