Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub's Metal Cloud

githubengineering.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: GitHub's Metal Cloud

#31

Post by vidarh »

stephenr wrote: post #20

These lines seem odd to me, maybe it's just the wording: > [gPanel] Deploying DNS via Heaven... > hubot is deploying dns/master (deadbeef) to production. > hubot's production deployment of dns/master (deadbeef) is done! (6s) Is this just an IMO odd use of the word "deploying" or does a DNS change really mean building and deploying a new package/image?

I deploy public DNS via new images. Image builds are fast, and our DNS changes rarely. When DNS changes frequently I wouldn't recommend it, though. Our internal DNS is using SkyDNS2 (backed by Etcd) instead, because that changes frequently (service registrations etc. as vms are started/stopped). But for the public DNS we like having the one DNS change => one git commit => one Docker image mapping to see who/why/when DNS changed.

Re: GitHub's Metal Cloud

#32

Post by ymse »

Hardware provisioning is a dying art. I would love to see a modern-day xCAT[0] clone that's easy to install and configure and with proper multi-platform support. Foreman is half-way there, but AFAIK doesn't do BMC provisioning and discovery, which is a big deal.

0: https://github.com/xcat2/xcat-core/blob/master/docs/source/i...

Re: GitHub's Metal Cloud

#33

Post by glenngillen »

jon-wood wrote: post #26
castell wrote: post #21

The Hubot workflow sounds interesting. It seems more and more DevOps prefer it. Has someone first hand experience with such Hubot usage? Do you prefer such commands or would you want to write more informal short sentences?

I'm somewhat split. There's definite value in having a shared history of what people have done, but I prefer that to take the form of command line tools pushing status updates to Hipchat or whatever. You lose so much convineance by pretending Hipchat's chat box is a terminal, everything from command history to being able to quickly iterate over the contents of a file or set environment variables.

One of the most compelling aspects of it for me isn't so much the shared history, but just visibility of what people have done. It's a really effective way of transferring knowledge of how things are done. You can easily drop into the room and watch play-by-play how a given task is done.

Re: GitHub's Metal Cloud

#34

Post by defect »

mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Also checkout https://github.com/Tumblr/genesis which does sort of the same thing as the PXE image GitHub writes about.

Re: GitHub's Metal Cloud

#35

Post by mariusmg »

lwhalen wrote: post #9
sargun wrote: post #2

This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.

Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all , and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and s…

>Sometimes you have a workload that really isn't a fit for virtualization

Yeah, any serious I/O load is unsuited for virtualization.

Re: GitHub's Metal Cloud

#36

Post by ktt »

castell wrote: post #21

The Hubot workflow sounds interesting. It seems more and more DevOps prefer it. Has someone first hand experience with such Hubot usage? Do you prefer such commands or would you want to write more informal short sentences?

Unfortunately most of the material on ChatOps currently covers only how to get Hubot to display cat pictures or other trivia [1]. Maybe it's because each company should create their own "chat API" but I'd also like to hear some real, inside "war stories".

Does anyone knows what app does GitHub use for chats? Looks like a simple and elegant UI over Basecamp.

[1]: http://hubot-script-catalog.herokuapp.com/

Re: GitHub's Metal Cloud

#37

Post by voltagex_ »

mverwijs wrote: post #11

We did something similar at Optiver. * boot a custom live cd (a la Knoppix) over PXE * Live CD places node into database if it doesn't exist yet, using dmidecode to find serialnumbers and such * Live Cd keeps querying database for instructions * Engineer adds a profile to the node in the database * Live Cd slices up disk to match the profile * Live Cd fetches a tarball of the base OS from an URL and throws it on the…

Hey cool so what I'm fiddling around with in iPXE isn't completely obsolete in the age of Docker and disposable VMs.

Re: GitHub's Metal Cloud

#38

Post by devonkim »

mariusmg wrote: post #35
lwhalen wrote: post #9

Earlier quoted context omitted.

Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all , and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and s…

>Sometimes you have a workload that really isn't a fit for virtualization Yeah, any serious I/O load is unsuited for virtualization.

A lot of other latency-sensitive applications tend to have so many adverse performance conditions (that can usually be remediated with a lot of blood sweat and tears) under virtualization that it becomes easier to just go bare metal and deal with physical infrastructure overhead.

Re: GitHub's Metal Cloud

#40

Post by mverwijs »

iofj wrote: post #14
mverwijs wrote: post #12

Earlier quoted context omitted.

I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!

The big issue I'm having with that is that it involves trusting vendors to get network boot right. Especially when it comes to the looping part of "loop until DHCP gets a response" it becomes a problem. One of the cheap vendor tries 30 times and then goes to a boot failed screen after trying the disk. Also, 1 time out of a 4-5000 or so network boot fails. Not sure why.

That's where iLO comes in. iLO is horrible, but you can ssh to it and set all manner of stuff.

When we didn't have PXE, we had a script that told iLO to boot from CD, and that the CD was located at http://something/bootme.iso. iLO would always have network, and would pass the .iso magically to the server as device to boot from.

Post reply on HN