Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub's Metal Cloud

githubengineering.com

Post reply on HN
Print view View original

71–80 of 83 posts

Re: GitHub's Metal Cloud

#71

Post by FireBeyond »

zeveb wrote: post #67
q3k wrote: post #61

Earlier quoted context omitted.

Newer packages, sane LTS policy, easier to get non-free firmware/drivers going (as in, the default CD comes with them), seemingly more support from third parties. They're both pretty awful due to their automagic(al tendency to break down in mysterious ways), but if I have to choose, I'll go with Ubuntu. Disclaimer: My main box runs Gentoo and I own no Mac machines, if that changes anything in your vision of Ubuntu us…

> Newer packages, sane LTS policy Those two are in opposition: Debian (generally) has new-enough packages, but it's stable , which is what one wants on a server system. Meanwhile, Debian's LTS story is better than Ubuntu's: just upgrade, and know it will work. > easier to get non-free firmware/drivers But how often is that needed for server systems? And of course, there're the ethical & engineering issues with using…

"stable". "new-enough"

This is a leading word. As is a lot of that paragraph. For many tools some companies use, Debian certainly is NOT "new-enough" with many package choices. Nor is Ubuntu inherently NOT "stable" - and still trails a little behind the leading edge. As for upgrades, I've watched many a server upgrade seamlessly from 10.04 LTS to 12.04 to 14.04. I'm sure there can be and has been many a person, many a thread who've not had seamless experiences. But the same applies for Debian - heck, even the release manual has a section entitled "How To Recover A Broken System" with reference to system upgrades.

"non-free firmware/drivers"

How often needed? In this article alone, IPMI, BMC, RAID, BIOS.

"And of course, there're the ethical & engineering issues"

This is a derailment. What exactly are the ethical issues for a closed source company in using other proprietary software?

I'm by no means an Ubuntu fanatic. It has its share of issues, absolutely. I have everything from FreeBSD to Debian to RHEL to OmniOS to administer, and they all have strengths and weaknesses.

Re: GitHub's Metal Cloud

#72

Post by scott_karana »

q3k wrote: post #61
wtbob wrote: post #50

Earlier quoted context omitted.

I too would be interested in the answer. From my perspective, Debian is the server Linux distro par excellence , and in my experience the folks who choose Ubuntu have been devs who don't actually use Linux (e.g., the sorts who develop on a Mac or in a VM rather than on a personal Linux system). It's not really fair to Ubuntu, which is decent enough in its own way, but I tend to consider the choice of Ubuntu to be a b…

Newer packages, sane LTS policy, easier to get non-free firmware/drivers going (as in, the default CD comes with them), seemingly more support from third parties. They're both pretty awful due to their automagic(al tendency to break down in mysterious ways), but if I have to choose, I'll go with Ubuntu. Disclaimer: My main box runs Gentoo and I own no Mac machines, if that changes anything in your vision of Ubuntu us…

Drivers are definitely a big deal on desktops, but for servers?...

Re: GitHub's Metal Cloud

#73

Post by stephenr »

dmourati wrote: post #64
A_Beer_Clinked wrote: post #19

Can anybody with experience of using Openstack Ironic[1] in this space comment on advantages of rolling your own liek GitHub? [1] https://wiki.openstack.org/wiki/Ironic

A consultant from Ubuntu called Ironic a "still birth" and said its name was indicative of its fit with the rest of OpenStack. His team used MaaS which I gather serves most of the same purpose: https://maas.ubuntu.com/

Breaking news: A consultant for a company trash talks an open source competitor.

Re: GitHub's Metal Cloud

#74

Post by otterley »

ptype wrote: post #49

Why would a company like GitHub choose Ubuntu over Debian? The LTS policy?

A better question, in my mind, is why they'd choose Ubuntu over CentOS or RHEL, since they're running on Dell hardware, and Kickstart is far, far superior to Debian/Ubuntu's PXE install (and preseeding is awful to configure). My org went through a lot of pain because of this choice (which preceded me) and I'd hate for anyone to go through what I went through.

Also Dell's maintenance tooling barely works on Ubuntu at all; they don't even officially support their OpenManage stack on it. And forget about online firmware upgrades.

Re: GitHub's Metal Cloud

#75

Post by SEJeff »

FireBeyond wrote: post #69
SEJeff wrote: post #54

Earlier quoted context omitted.

Vs just buying opengear console servers and something like conserver to get the actual text via serial like most large Unix environments (every job I've worked at) do. Then you can just scrape the text.

Right, this surprised me. They're doing IPMI, so why not SOL (Serial over LAN) to get the raw stream? Actually, on second thoughts, I'm not surprised so much. I've read many threads on Supermicro IPMI and people's frustration with it (reliant on outdated Java, and hacked together wrappers over VNC) that make it seem like a deliberate choice to obfuscate things -just- enough to make other tools difficult.

No a career building those types of tools taught me SoL is garbage from most vendors. Cray, Dell, and HP are (arguably) best with mostly reliable SoL, but they still are awful. If you paste a buffer too big into a SoL session, the dell DRAC will freeze, so you have to kill and restart the serial connection. If you have > 1000 machines, hardware serial is the best thing to do for management, in addition to IPMI for power management.

Re: GitHub's Metal Cloud

#76

Post by amalag »

FireBeyond wrote: post #70
amalag wrote: post #66

Is this also done remotely by gPanel? >Once we've gathered all the information we need about the machine, we enter configuring, where we assign a static IP address to the IPMI interface and tweak our BIOS settings. From there we move to firmware_upgrade where we update FCB, BMC, BIOS, RAID, and any other firmware we'd like to manage on the system.

In theory it should be if you have a tightly controlled hardware process (and in this case, Dell, who is used to selling servers configured to initially PXE boot, etc), and you have some 'expect/send' scripting in place.

I was not aware you can control BIOS settings, BIOS upgrades and configuring IPMI remotely like that.

EDIT - looks like it is straightforward if you control the IPMI locally. So the software would send commands to do it locally.

Re: GitHub's Metal Cloud

#77

Post by voltagex_ »

mverwijs wrote: post #42
voltagex_ wrote: post #37

Earlier quoted context omitted.

Hey cool so what I'm fiddling around with in iPXE isn't completely obsolete in the age of Docker and disposable VMs.

Well, no. But IMHO, iron is only do-able if you have enough of it. This I learned the hard way at my previous contract, where we only had a handful of servers and all of them production. When you want to automate your complete infra, including rolling out hardware, you need hardware to develop and test on. Entropy of life will ensure that exactly that moment you need to reinstall that PostgreSQL slave from scratch, t…

In a perfect world you'd have a Dev, Staging and Prod PXE server, but reality says you're not going to be able to get signoff to run that many.

Re: GitHub's Metal Cloud

#78

Post by mverwijs »

voltagex_ wrote: post #77
mverwijs wrote: post #42

Earlier quoted context omitted.

Well, no. But IMHO, iron is only do-able if you have enough of it. This I learned the hard way at my previous contract, where we only had a handful of servers and all of them production. When you want to automate your complete infra, including rolling out hardware, you need hardware to develop and test on. Entropy of life will ensure that exactly that moment you need to reinstall that PostgreSQL slave from scratch, t…

In a perfect world you'd have a Dev, Staging and Prod PXE server, but reality says you're not going to be able to get signoff to run that many.

But that is perfectly fine! That only means that you cannot have an automated install procedure that the company can rely on. There is really nothing wrong with a little manual labour at this stage. Do not spend weeks and weeks on automating this without having the environment to test your code.

Re: GitHub's Metal Cloud

#79

Post by stephenr »

otterley wrote: post #74
ptype wrote: post #49

Why would a company like GitHub choose Ubuntu over Debian? The LTS policy?

A better question, in my mind, is why they'd choose Ubuntu over CentOS or RHEL, since they're running on Dell hardware, and Kickstart is far, far superior to Debian/Ubuntu's PXE install (and preseeding is awful to configure). My org went through a lot of pain because of this choice (which preceded me) and I'd hate for anyone to go through what I went through. Also Dell's maintenance tooling barely works on Ubuntu at…

> why they'd choose Ubuntu over CentOS or RHEL

Maybe they'd actually used them before.

CentOS is a turd of a distribution, I'm certain the only reason it has any marketshare is because it's the only supported "free" OS for cPanel/WHM which a lot of web hosts provide for non-technical customers.

Re: GitHub's Metal Cloud

#80

Post by Jgrubb »

stephenr wrote: post #79
otterley wrote: post #74

Earlier quoted context omitted.

A better question, in my mind, is why they'd choose Ubuntu over CentOS or RHEL, since they're running on Dell hardware, and Kickstart is far, far superior to Debian/Ubuntu's PXE install (and preseeding is awful to configure). My org went through a lot of pain because of this choice (which preceded me) and I'd hate for anyone to go through what I went through. Also Dell's maintenance tooling barely works on Ubuntu at…

> why they'd choose Ubuntu over CentOS or RHEL Maybe they'd actually used them before. CentOS is a turd of a distribution, I'm certain the only reason it has any marketshare is because it's the only supported "free" OS for cPanel/WHM which a lot of web hosts provide for non-technical customers.

I personally think it gets share with hosts because it never updates anything so it's less work for them to maintain (which I can't hate on).
Post reply on HN