The Hubot workflow sounds interesting. It seems more and more DevOps prefer it.
Has someone first hand experience with such Hubot usage? Do you prefer such commands or would you want to write more informal short sentences?
GitHub's Metal Cloud
21–30 of 83 posts
Re: GitHub's Metal Cloud
#22The Hubot workflow sounds interesting. It seems more and more DevOps prefer it. Has someone first hand experience with such Hubot usage? Do you prefer such commands or would you want to write more informal short sentences?
There was also a talk titled "Chatops" about this specific thing if you want to learn more: https://www.youtube.com/watch?v=NST3u-GjjFw
Re: GitHub's Metal Cloud
#23These lines seem odd to me, maybe it's just the wording: > [gPanel] Deploying DNS via Heaven... > hubot is deploying dns/master (deadbeef) to production. > hubot's production deployment of dns/master (deadbeef) is done! (6s) Is this just an IMO odd use of the word "deploying" or does a DNS change really mean building and deploying a new package/image?
I've always thought of 'deploy' as a generic term for pushing any change to production. I think this is pretty typical - e.g. you deploy your application, even if that just means updating some files.
Re: GitHub's Metal Cloud
#24These lines seem odd to me, maybe it's just the wording: > [gPanel] Deploying DNS via Heaven... > hubot is deploying dns/master (deadbeef) to production. > hubot's production deployment of dns/master (deadbeef) is done! (6s) Is this just an IMO odd use of the word "deploying" or does a DNS change really mean building and deploying a new package/image?
They probably manage DNS in a Git repository/using Puppet, so deploying may be quite literal. I see no issue with that.
Re: GitHub's Metal Cloud
#25These lines seem odd to me, maybe it's just the wording: > [gPanel] Deploying DNS via Heaven... > hubot is deploying dns/master (deadbeef) to production. > hubot's production deployment of dns/master (deadbeef) is done! (6s) Is this just an IMO odd use of the word "deploying" or does a DNS change really mean building and deploying a new package/image?
They probably manage DNS in a Git repository/using Puppet, so deploying may be quite literal. I see no issue with that.
We do this, it works well do us with 200-300 servers.
Re: GitHub's Metal Cloud
#26The Hubot workflow sounds interesting. It seems more and more DevOps prefer it. Has someone first hand experience with such Hubot usage? Do you prefer such commands or would you want to write more informal short sentences?
I'm somewhat split. There's definite value in having a shared history of what people have done, but I prefer that to take the form of command line tools pushing status updates to Hipchat or whatever. You lose so much convineance by pretending Hipchat's chat box is a terminal, everything from command history to being able to quickly iterate over the contents of a file or set environment variables.
Re: GitHub's Metal Cloud
#27Can anybody with experience of using Openstack Ironic[1] in this space comment on advantages of rolling your own liek GitHub? [1] https://wiki.openstack.org/wiki/Ironic
If you are not running OpenStack otherwise I'd say maintaining it is way to much effort to get relatively basic functionality. They also would still need custom work for asset databases, the memory checks, so they don't gain much.
Re: GitHub's Metal Cloud
#28This seems wrong to me. This was the state of the art ~3 years ago. Now, I feel like all of the machines should be provisioned already with an OS, and a basic image, and a orchestration system like CoreOS / Mesos / Docker should specialize them. IMHO, requiring hardware, or the entire machine should be exception, not the rule.
I built something similar for a managed hosting provider ~10 years ago. That doesn't make it any less useful now, and this does a ton more than our tool did, far more elegantly.
At some point someone needs to manage the actual hardware, whether that's you, or a middleman, and when you're handling hundreds or even thousands of devices its just not practical without automation.
Re: GitHub's Metal Cloud
#29Earlier quoted context omitted.
I'm surprised everyone is still installing to disk. In 2008/2009, we had a POC where we ran the OS from memory after booting over PXE. * boot a live image into memory * point LXD/RKT/Docker to /containers * ... * profit!
The big issue I'm having with that is that it involves trusting vendors to get network boot right. Especially when it comes to the looping part of "loop until DHCP gets a response" it becomes a problem. One of the cheap vendor tries 30 times and then goes to a boot failed screen after trying the disk. Also, 1 time out of a 4-5000 or so network boot fails. Not sure why.
If you have IPMI on the server this doesn't become such a big problem - you can reasonably trigger resets/reboots if it's not up after a given amount of time.
Re: GitHub's Metal Cloud
#30Earlier quoted context omitted.
Sometimes you have a workload that really isn't a fit for virtualization/containers/whatever the latest Rails hotness is, at all , and you just need to throw a couple of cargo trailers of insanely massively-spec'd servers at the problem. In those cases, your 'old school' server provisioning toolkit had better be on-point. It's easy to forget just how ridiculously powerful bare iron is these days. Go to Dell.com and s…
Containers are just a namespacing tool, though; you're still running on bare metal (well, bare Linux). Docker in particular runs on AuFS, which is slow, but other containerization tools just use a chroot.
Docker can run on any number of things, include btrfs and overlayfs+ext4, as well as devicemapper. E.g. CoreOS defaults to overlayfs.