Server BMCs can need to be rebooted every so often
utcc.utoronto.ca
Server BMCs can need to be rebooted every so often
1–10 of 96 posts
Re: Server BMCs can need to be rebooted every so often
#2This is brought to you by the BMC with a KVM-over-IP that wouldn't accept '2' entered on the (virtual) keyboard in any way or form.
This is one of those things that I'd probably be willing to spend a ton of time analysing to find the root cause if I could. Dump the memory image, debug the code, and figure out why it's just that (and possibly other) keys. My guess just from the description is that a bitflip happened exactly to the entry of a character map translation table or similar.
Re: Server BMCs can need to be rebooted every so often
#3On a Sun box we had, the system controller would panic the domains every so many days (I think ~700).
You could have rebooted the domains in the chassis for regular patches, but if you hadn't restarted the SC, you were in for a surprise.
Restart your management devices!
Re: Server BMCs can need to be rebooted every so often
#4BMC: Baseboard Management Controller
Re: Server BMCs can need to be rebooted every so often
#5On a Sun box we had, the system controller would panic the domains every so many days (I think ~700). You could have rebooted the domains in the chassis for regular patches, but if you hadn't restarted the SC, you were in for a surprise. Restart your management devices!
Yes I remember something similar on Sun Fire 6800.
Another issue was a firmware bug on the Sun Netra X1 where rebooting or updating the lom would result in a reset of the host. Not fun with UFS without logging enabled.
Re: Server BMCs can need to be rebooted every so often
#6There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc
And the worst are those that use java applets or webstart and require a ancient java version.
Re: Server BMCs can need to be rebooted every so often
#7There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.
Looking at you, Asrock...
Re: Server BMCs can need to be rebooted every so often
#8There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.
I am honestly surprised how bad many of these are, and in production, no less.
I recently set up a supermicro system and spent a whole day just trying to figure out what to install to get the stupid ancient Java crap to load so I could mount an ISO.
Re: Server BMCs can need to be rebooted every so often
#9Someone once told me that nobody ever returned a server because the BMC sucked too much, and that says everything you need to know about BMCs.
Incidentally, at a previous gig, the software that handled issuing BMC commands had a BMC reboot as the first step in any script it ran.
Re: Server BMCs can need to be rebooted every so often
#10Anecdotally, this reminds me of a terrible experience with some Dell hardware back around 2011. I volunteered to do an installation of some hardware in a POP cage at one of the London UK colos.
Hardware arrived a day or two late, got it unpacked and carted upstairs. Spent a few days installing it (had to build a deployment environment on my Mac in a VM because the servers had no CDROM drive, only NetBoot and USB; our centralised imaging system hadn't been built with "installs outside of core DC network" in mind).
Discovered that the machines had been ordered with iDRACs, not DRACs. Damn things did not work until the box was booted into Linux and a driver to interface/activate the board was loaded. This, of course, worked wonderfully when the kernel locked up with a panic.
The machines were being used to test custom kernel modifications. Panics and lockups were common. Every time it happened, the engineers in the US had to ask "smart hands" to go reset the machine in the cage.
Experiment was a success though. Turned into the foundation for how a very large edge network was built.