Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Server BMCs can need to be rebooted every so often

utcc.utoronto.ca

Post reply on HN
Print view View original

81–90 of 96 posts

Re: Server BMCs can need to be rebooted every so often

#81

Post by rbanffy »

matja wrote: post #49
fulafel wrote: post #45

People are surprisingly chill about BMCs as attack surface, eg Intel management features get a lot more scrutiny.

IMO because you can configure the BMC to use it's own dedicated NIC and network. That network should have no routing to the Internet and no production traffic, just BMC management. When Intel ME shares the NIC used for the OS traffic - that is far harder to secure.

> That network should have no routing to the Internet and no production traffic, just BMC management.

And no single computer with NICs on both the BMC network and one that can access the internet, hopefully.

All it takes is one single mistake.

Re: Server BMCs can need to be rebooted every so often

#82

Post by xoa »

toast0 wrote: post #65
xoa wrote: post #44

Earlier quoted context omitted.

This was me though I ended up moving to SuperMicro instead. Not fancy but their BMC seems to get the job done, and also doesn't cost a significant chunk of money extra for basic functionality. And then even more money to have IPMI be dedicated connection not shared. And HP's frigging BIOS wouldn't even work with their own HP rack console! But it was happy with an Apple mouse and keyboard. Argh! Making me irritated ag…

> SuperMicro instead. Not fancy but their BMC seems to get the job done Kind of depends which generation of servers. I had worked with a lot of x9 and x10 (xeon e-2600v1-4) which was alright, as long as you don't mind outdated java (well the newest x10 bmcs do html5 consoles too, IIRC); but I recently started renting an x8 server personally, and it's worse... My favorite is when serial over lan just stops responding…

Totally fair. Yeah should have specified I'm talking fairly new stuff, at least new enough it all has HTML5 consoles and mildly more polish. Not that there aren't problems, less a "bug" then a "cutting edge for newbies" it does near zilch certificate validation for example. So when the friendly new guy wasn't paying attention and generated a client instead of server certificate and uploaded it to a remote one it cheerfully accepted it and whoops! Operator error of course but the real point was that remote rectifying it proved surprisingly impossible despite still having admin serial/ssh access. Documentation is bad and didn't seem to like typical reset codes or tools.

I guess after lots of problems with expensive "high end" fancy stuff like HPE's and co I kind of felt resigned that all BMC/IPMI was kind of crap, and at least with SM's I felt less obviously squeezed. Like I said math will all be different for high end stuff and big herds. But neither Dell or HPE's struck me as "oh yeah that's worth an extra $300 on every single last server!"

Re: Server BMCs can need to be rebooted every so often

#83

Post by flemhans »

dsr_ wrote: post #52
sweetjuly wrote: post #32

Earlier quoted context omitted.

I wonder what the BMC was doing to hurt performance. Maybe it was questionable DVFS that stopped when the BMC died? Poor fan management that caused thermal throttling?

That's a pretty good guess: lots of BMCs have fan control, and turning it off should make all fans stay at max power. In a datacenter the odds of anyone noticing that the fans are always on high is practically nil, unless you are specifically monitoring fan RPM. Most folks don't bother, as what you are actually interested in is temperature.

Really high RPM fans may also stand out audibly in the data center

Re: Server BMCs can need to be rebooted every so often

#84

Post by sys_64738 »

fulafel wrote: post #77
sys_64738 wrote: post #74

Earlier quoted context omitted.

You should never keep the BMC plugged into a network. Security 101.

That view is what I meant by my comment. IMO it's wrong - It's terrible product design that servers have network ports that will cause catastrophic failures unless carefully only connected to a special expertly secured un-network for fragile things.

It’s not wrong regardless of the BMC security. You don’t plug things into the network that don’t need to be plugged in. BMCs only need network access for servicing most cases.

Re: Server BMCs can need to be rebooted every so often

#85

Post by p_l »

andrewjf wrote: post #71
p_l wrote: post #43

Earlier quoted context omitted.

This is one of the reasons I like to architect networks for netbooting (so no remote media needed) plus force every physical server to boot UEFI-only - because UEFI supports serial console properly, unlike BIOS, so I can just use IPMI Serial-over-LAN support. Combination of those two generally removes the need for any of the advanced features that required custom clients or even a Web browser

For sure - we have some networks that when a host netboots it always goes to something like http://netboot.xyz with serial console by default. My favorite is some vendors using COM1 and some using COM2 so you have no idea which it is ahead of time.

That's why I keep to UEFI - most of the time the configuration just works, and firmware passes down information about serial console to the OS, iirc.

Re: Server BMCs can need to be rebooted every so often

#86

Post by BenjiWiebe »

sys_64738 wrote: post #73

> Server BMCs are little computers running ancient versions of Linux with software that's probably terribly written and they stay running forever, which means all sorts of opportunities for slow bugs. Reboot away! Such a silly comment. If your BMC is updated then it will use a recent version of Linux.

I'm curious why you say that. It's not a given that updating something (especially embedded!) will actually bring it to a modern version.

You can install an update for a router and end up with an absolutely ancient Linux kernel on it.

Re: Server BMCs can need to be rebooted every so often

#87

Post by MertsA »

gjvc wrote: post #37

This is why staggered reboots of stuff, weekly or monthly, avoids this class of problem. It's simple and some might say it dumbs down the role of infrastructure management, but it sure as hell beats the feeling in the middle of the workday/workweek ... "It's lost grip. NFI what to do. Can't see anything. Reboot it FFS."

Bold of you to assume a regular reboot will also reboot the BMC (it won't although I guess there might be BMCs which do). Some things you really need a full cold boot. I've seen a test cluster of storage servers where after rebooting the whole cluster all in one go enough failed to boot that data was unavailable until a few servers were fixed due to flaky RAM that failed to make it past memory training on boot but was "fine" as far as we could tell until we rebooted.

I'd mostly agree with you, but this isn't always as simple as it may seem to "just reboot it" and there can be subtle differences between what you're exercising with your rolling reboots and what actually happens in a real complete power loss scenario. Plenty of stuff can break and you'll have no idea until you're trying to get stuff back up after a power loss event and you're up a creek.

Re: Server BMCs can need to be rebooted every so often

#88

Post by klysm »

lapinot wrote: post #78
klysm wrote: post #63

Earlier quoted context omitted.

I was looking into this recently. It doesn’t seem like you can easily get your hands on BMC hardware. There is one project I found where they’re using an FPGA and everything is open source but it still looked far from easy.

Afaik the problem is typically not hardware but software, since most server motherboards come with a bmc chip on it. Are you designing a motherboard? All i would ask for a bmc is root access. But now everyone (but the big players, who design their own motherboards) is basically stuck with a stupid embedded linux with shitty software that has half-backed features they don't even need, but now need to care about. When…

I was interested in building a board from scratch with a BMC on it

Re: Server BMCs can need to be rebooted every so often

#89

Post by deadfece »

dveeden2 wrote: post #5
deadfece wrote: post #3

On a Sun box we had, the system controller would panic the domains every so many days (I think ~700). You could have rebooted the domains in the chassis for regular patches, but if you hadn't restarted the SC, you were in for a surprise. Restart your management devices!

Yes I remember something similar on Sun Fire 6800. Another issue was a firmware bug on the Sun Netra X1 where rebooting or updating the lom would result in a reset of the host. Not fun with UFS without logging enabled.

I can't remember which direction this went, but I had a Netra T1 hooked to another Sun machine (240R? V440?, don't remember), and resetting one would send a break out on the serial console... which would send the other into the ok> prompt in OBP.

We finally got serial servers out of that one, though!

Re: Server BMCs can need to be rebooted every so often

#90

Post by jebr224 »

rbanffy wrote: post #79
lapinot wrote: post #78

Earlier quoted context omitted.

Afaik the problem is typically not hardware but software, since most server motherboards come with a bmc chip on it. Are you designing a motherboard? All i would ask for a bmc is root access. But now everyone (but the big players, who design their own motherboards) is basically stuck with a stupid embedded linux with shitty software that has half-backed features they don't even need, but now need to care about. When…

We got into that when we started expecting servers to have VGA and PS/2 ports because an RS-232 console would not be very useful with a GUI. Is there a modular standard for BMCs?

There are two standards for pluggable BMC's

  - runbmc (https://www.opencompute.org/documents/ocp-runbmc-daughterboard-card-design-specification-v1-4-1-pdf)

  - DC-SCM https://www.opencompute.org/documents/ocp-dc-scm-spec-rev-1-0-pdf
I have only glanced the surface of these specs.
Post reply on HN