Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Server BMCs can need to be rebooted every so often

utcc.utoronto.ca

Post reply on HN
Print view View original

91–96 of 96 posts

Re: Server BMCs can need to be rebooted every so often

#91

Post by jebr224 »

lapinot wrote: post #78
klysm wrote: post #63

Earlier quoted context omitted.

I was looking into this recently. It doesn’t seem like you can easily get your hands on BMC hardware. There is one project I found where they’re using an FPGA and everything is open source but it still looked far from easy.

Afaik the problem is typically not hardware but software, since most server motherboards come with a bmc chip on it. Are you designing a motherboard? All i would ask for a bmc is root access. But now everyone (but the big players, who design their own motherboards) is basically stuck with a stupid embedded linux with shitty software that has half-backed features they don't even need, but now need to care about. When…

> Are you designing a motherboard? Yes, the team I work for designs server motherboards.

BMC security is what keeps me up at night. Firmware software quality is low, and often not up to date. I think openbmc does a good job in both respects.

Re: Server BMCs can need to be rebooted every so often

#92

Post by [deleted] »

MertsA wrote: post #87
gjvc wrote: post #37

This is why staggered reboots of stuff, weekly or monthly, avoids this class of problem. It's simple and some might say it dumbs down the role of infrastructure management, but it sure as hell beats the feeling in the middle of the workday/workweek ... "It's lost grip. NFI what to do. Can't see anything. Reboot it FFS."

Bold of you to assume a regular reboot will also reboot the BMC (it won't although I guess there might be BMCs which do). Some things you really need a full cold boot. I've seen a test cluster of storage servers where after rebooting the whole cluster all in one go enough failed to boot that data was unavailable until a few servers were fixed due to flaky RAM that failed to make it past memory training on boot but wa…

[deleted]

Re: Server BMCs can need to be rebooted every so often

#93

Post by 1amzave »

amluto wrote: post #30
jebr224 wrote: post #28

I do development on openbmc ( https://github.com/openbmc/openbmc ) which is an open source bmc implantation using bitbake, primarily targeted to aspeed, and nuvoton bmc chips.

Can a mere mortal usefully deploy openbmc on a small number of servers?

If the servers in question happen to be on the list of supported hardware, quite possibly. (I don't know of any up-to-date online list, but running `source setup` in the root of the source tree will print it.)

Re: Server BMCs can need to be rebooted every so often

#94

Post by ixs »

toast0 wrote: post #65
xoa wrote: post #44

Earlier quoted context omitted.

This was me though I ended up moving to SuperMicro instead. Not fancy but their BMC seems to get the job done, and also doesn't cost a significant chunk of money extra for basic functionality. And then even more money to have IPMI be dedicated connection not shared. And HP's frigging BIOS wouldn't even work with their own HP rack console! But it was happy with an Apple mouse and keyboard. Argh! Making me irritated ag…

> SuperMicro instead. Not fancy but their BMC seems to get the job done Kind of depends which generation of servers. I had worked with a lot of x9 and x10 (xeon e-2600v1-4) which was alright, as long as you don't mind outdated java (well the newest x10 bmcs do html5 consoles too, IIRC); but I recently started renting an x8 server personally, and it's worse... My favorite is when serial over lan just stops responding…

I was just fiddling around with a SuperMicro X8 IPMI the other day. The X8 IPMI stuff is terrible, e.g. the warning that your Java installation is outdated on opening the website etc.

Turns out, you can actually install X9 IPMI firmware on X8 boards as the platform files are still shipped. Might be worth checking out, if this improves things for you. It did for me.

Check out https://github.com/devicenull/ipmi_firmware_tools for unpacking (and repacking) the SuperMicro firmware. The developer just merged my patches making it work with some of the X8 boards. As long as your board is listed in /etc/defaults of the IPMI tree you should be good.

Re: Server BMCs can need to be rebooted every so often

#95

Post by ixs »

kube-system wrote: post #8
dveeden2 wrote: post #6

There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.

I am honestly surprised how bad many of these are, and in production, no less. I recently set up a supermicro system and spent a whole day just trying to figure out what to install to get the stupid ancient Java crap to load so I could mount an ISO.

There are reasons for tools like https://github.com/ixs/kvm-cli. It seems like every operations team built their own version that logs into the web interfaces, downloads the java stuff and then runs it locally...

Just annoying.

Re: Server BMCs can need to be rebooted every so often

#96

Post by ixs »

arcticgeek wrote: post #21
dveeden2 wrote: post #17

Earlier quoted context omitted.

With current browsers Java applets are not supported anymore. Some older HPE systems the didn't update the firmware to provide alternatives. I've seen multiple vendors with problematic code that didn't work with newer Java versions. This is by no means meant to bash Java. Some non-Java BMCs can be horrible as well (e.g. require many TCP ports in a firewall/tunnel unfriendly way or require SSH with old algorithms that…

I‘ve really only had experience with Supermicro BMCs but I totally believe you that there are lots of crufty OOB environments in the wild which are hard to work with. While it’s true that applets don’t work anymore (probably a good thing), and therefore the experience isn‘t as integrated or seemless as it once was, it’s a practical matter to just log into the BMC and click on the console preview, using the JNLP file…

For fun: This actually runs the Java Applet KVM viewer on a SuperMicro X7 board: https://github.com/ixs/kvm-cli/blob/master/kvm_x7.py

1. Downloads the data from the IPMI interface

2. Modifies the files to run locally

3. Writes out a Java configuration with weak security settings so that TLS works with the deprecated ciphers.

4. Fires off a socat instance to redirect the localhost ports to the remote IPMI device.

5. Starts appletviewer locally.

Great fun writing that. Thank god we decomissioned the last X7 based storage appliances a while ago...

Post reply on HN