Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Server BMCs can need to be rebooted every so often

utcc.utoronto.ca

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Server BMCs can need to be rebooted every so often

#41

Post by gnufx »

isopede wrote: post #18

A few of my recent systems have come with a built-in combo BMC on the motherboard NICs. I haven't seen this before, is it a new trend? I'm imagining they put in a switch in there with the NICs? How does this even work? I'm doing a few timing sensitive projects involving hardware timestamping in the NICs. Does this mess with say timing variability? I've disabled the BMCs out of paranoia but I don't know the topology i…

That's common, and something to avoid. You want a physically separate BMC network for security and reliability.

Re: Server BMCs can need to be rebooted every so often

#42

Post by Peanuts99 »

maltris wrote: post #13

Experiences with iDRAC 6, 7, 8 have been terrible. After high runtime they stop to respond via HTTP and SNMP, its just rubbish. Reboot of BMC helped Sometimes, sometimes only a powerdrain. Back then even the ProSupport could not do much, they support a system unsupported by their devs. The latest iDRACs (with the fancy blue GUI) work a little better. I have no numbers to back all that, its just a feeling, maybe the p…

My feeling mirrors yours about the latest iteration of iDRAC, we've installed stacks of Dell servers recently and not had any issues at all.

Re: Server BMCs can need to be rebooted every so often

#43

Post by p_l »

dveeden2 wrote: post #6

There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.

This is one of the reasons I like to architect networks for netbooting (so no remote media needed) plus force every physical server to boot UEFI-only - because UEFI supports serial console properly, unlike BIOS, so I can just use IPMI Serial-over-LAN support.

Combination of those two generally removes the need for any of the advanced features that required custom clients or even a Web browser

Re: Server BMCs can need to be rebooted every so often

#44

Post by xoa »

themoonisachees wrote: post #26
roughly wrote: post #9

Someone once told me that nobody ever returned a server because the BMC sucked too much, and that says everything you need to know about BMCs. Incidentally, at a previous gig, the software that handled issuing BMC commands had a BMC reboot as the first step in any script it ran.

Not returned, but i've seen clients refuse to go with HP again because iLO sucked too much and come upgrade time went Dell. Unfortunately for them right around the time iLO got okay-ish and iDRAC got shitty.

This was me though I ended up moving to SuperMicro instead. Not fancy but their BMC seems to get the job done, and also doesn't cost a significant chunk of money extra for basic functionality. And then even more money to have IPMI be dedicated connection not shared. And HP's frigging BIOS wouldn't even work with their own HP rack console! But it was happy with an Apple mouse and keyboard. Argh! Making me irritated again remembering it. But yeah, we didn't return them but did sell them off to some other poor sucker and never bought an HP again. They may be great at large scale with the higher end central management software suite and support etc, but the BMC definitely was a dealbreaker at an SMB level. It's one of the few differentiating features vs a pile of other competitors who do the same basic thing, so I do think it's not entirely meaningless to get it badly wrong.

Re: Server BMCs can need to be rebooted every so often

#46

Post by Tijdreiziger »

somat wrote: post #35

As a big openbsd aficionado I always wish I could just run bog standard openbsd on my bmc's. ipmi does not do much for me so I would be fine if it were missing a good opensource ipmi stack. Most people would just want to run linux, but the point is, the bmc is just a small computer in charge of monitoring your big computer, however the bmc usually is unable to run an off the shelf operating system, so it is running a…

> An idea I have floating around in the back of my head is to just use a raspberry pi as a bmc on consumer grade hardware, I am sure it would turn out more complicated than this, but basically just hook the power button, i2c, and other headers to the pi's gpio. now you have a bmc that runs an off the shelf os.

You’re looking for one of the following projects:

https://pikvm.org/

https://tinypilotkvm.com/

Re: Server BMCs can need to be rebooted every so often

#47

Post by ilyt »

The worst bug I've seen in (IBM) BMC

The device in question didn't had NTP so to get accurate BMC logs timing it had to be synced from host OS

BMC is them fucking up the watchdog implementation, that worked on wall time.

Which means if you changed time in-between OS sending watchdog ping and BMC checking for timeout, you got system reboot.

But it was intermittent enough that you might set the cron to update the BMC clock (because you want accurate time on it for log correlation) and be lucky enough time to not connect to the cause when it finally happens.

Also we generally observed that BMCs got better, 10 years server were lucky if it was working and not crashing every few weeks (all mostly on IBM gear), to the point we had to stop alerting on that, and it was unreliable as method for OOB access, so far anything newer than few years have been fine. Althought I have no idea how you make modern hardware take minutes to return list of sensors(I'm looking at you Lenovo)

Re: Server BMCs can need to be rebooted every so often

#48

Post by ilyt »

fulafel wrote: post #45

People are surprisingly chill about BMCs as attack surface, eg Intel management features get a lot more scrutiny.

In proper environment the BMC is on its own dedicated NIC with no way to bridge to that network from the machine and only access from host machine being from root/admin account.

And then there are people that just port-forward BMC ports to the internet as cheap remote KVM...

Re: Server BMCs can need to be rebooted every so often

#49

Post by matja »

fulafel wrote: post #45

People are surprisingly chill about BMCs as attack surface, eg Intel management features get a lot more scrutiny.

IMO because you can configure the BMC to use it's own dedicated NIC and network.

That network should have no routing to the Internet and no production traffic, just BMC management.

When Intel ME shares the NIC used for the OS traffic - that is far harder to secure.

Re: Server BMCs can need to be rebooted every so often

#50

Post by ilyt »

xoa wrote: post #44
themoonisachees wrote: post #26

Earlier quoted context omitted.

Not returned, but i've seen clients refuse to go with HP again because iLO sucked too much and come upgrade time went Dell. Unfortunately for them right around the time iLO got okay-ish and iDRAC got shitty.

This was me though I ended up moving to SuperMicro instead. Not fancy but their BMC seems to get the job done, and also doesn't cost a significant chunk of money extra for basic functionality. And then even more money to have IPMI be dedicated connection not shared. And HP's frigging BIOS wouldn't even work with their own HP rack console! But it was happy with an Apple mouse and keyboard. Argh! Making me irritated ag…

That pisses me off, when some vendors want to charge extra for KVM on stuff that already have hardware to do it and competitors do it...
Post reply on HN