Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Server BMCs can need to be rebooted every so often

utcc.utoronto.ca

Post reply on HN
Print view View original

31–40 of 96 posts

Re: Server BMCs can need to be rebooted every so often

#31

Post by pid-1 »

dveeden2 wrote: post #6

There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.

My first job (2016) involved dealing lots of servers used by telecom providers in many different DCs in different cities.

One of my most useful tools was a thumb drive full of old internet explorer, firefox, and Java installers.

The 50m ethernet cable (nicknamed BFC - big fucking cable) also deserves an honorable mention.

Re: Server BMCs can need to be rebooted every so often

#32

Post by sweetjuly »

iforgotpassword wrote: post #20

A colleague from a different department was managing a fleet of servers doing a lot of computing. Their BMCs would just stop working after a few weeks of uptime, every time. He was annoyed by it but mostly ignored it as it was mostly just used for additional monitoring. The power outlets in the racks were remote controllable, so you could still hard reset a server if this was required. However, as this fleet was used…

I wonder what the BMC was doing to hurt performance. Maybe it was questionable DVFS that stopped when the BMC died? Poor fan management that caused thermal throttling?

Re: Server BMCs can need to be rebooted every so often

#33

Post by shenki0 »

amluto wrote: post #30
jebr224 wrote: post #28

I do development on openbmc ( https://github.com/openbmc/openbmc ) which is an open source bmc implantation using bitbake, primarily targeted to aspeed, and nuvoton bmc chips.

Can a mere mortal usefully deploy openbmc on a small number of servers?

Not really. You need support from the manufacturer.

Recent BMC firmwares have openbmc under the hood, with the industry using it as a base for further development or customisation.

Re: Server BMCs can need to be rebooted every so often

#34

Post by da768 »

dveeden2 wrote: post #6

There are too many names for the BMCs, even within a single vendor. BMC, ILO, LOM, DRAC, iDRAC, etc And the worst are those that use java applets or webstart and require a ancient java version.

Docker image with Firefox 52 ESR and Java plugin comes to the rescue.

Re: Server BMCs can need to be rebooted every so often

#35

Post by somat »

As a big openbsd aficionado I always wish I could just run bog standard openbsd on my bmc's. ipmi does not do much for me so I would be fine if it were missing a good opensource ipmi stack. Most people would just want to run linux, but the point is, the bmc is just a small computer in charge of monitoring your big computer, however the bmc usually is unable to run an off the shelf operating system, so it is running an old proprietary version of linux with proprietary software. this situation sucks.

An idea I have floating around in the back of my head is to just use a raspberry pi as a bmc on consumer grade hardware, I am sure it would turn out more complicated than this, but basically just hook the power button, i2c, and other headers to the pi's gpio. now you have a bmc that runs an off the shelf os.

Re: Server BMCs can need to be rebooted every so often

#36

Post by c0l0 »

iforgotpassword wrote: post #20

A colleague from a different department was managing a fleet of servers doing a lot of computing. Their BMCs would just stop working after a few weeks of uptime, every time. He was annoyed by it but mostly ignored it as it was mostly just used for additional monitoring. The power outlets in the racks were remote controllable, so you could still hard reset a server if this was required. However, as this fleet was used…

I've seen pretty much the opposite close to a decade ago with a number of AMD Opteron/15h-based servers we had, where the BMC could end up in a bad state that made the ipmi-driver-spawned kernel thread handling it burn a whole core for no good reason.

Re: Server BMCs can need to be rebooted every so often

#37

Post by gjvc »

This is why staggered reboots of stuff, weekly or monthly, avoids this class of problem. It's simple and some might say it dumbs down the role of infrastructure management, but it sure as hell beats the feeling in the middle of the workday/workweek ... "It's lost grip. NFI what to do. Can't see anything. Reboot it FFS."

Re: Server BMCs can need to be rebooted every so often

#38

Post by wil421 »

somat wrote: post #35

As a big openbsd aficionado I always wish I could just run bog standard openbsd on my bmc's. ipmi does not do much for me so I would be fine if it were missing a good opensource ipmi stack. Most people would just want to run linux, but the point is, the bmc is just a small computer in charge of monitoring your big computer, however the bmc usually is unable to run an off the shelf operating system, so it is running a…

I’ve heard of people using a Raspberry PI as a “BMC” by using VNC viewer connect. Here’s one example I found[1].

[1]https://forums.unraid.net/topic/97460-selfmade-ipmibmc-with-...

Re: Server BMCs can need to be rebooted every so often

#39

Post by yellow_lead »

userbinator wrote: post #2

This is brought to you by the BMC with a KVM-over-IP that wouldn't accept '2' entered on the (virtual) keyboard in any way or form. This is one of those things that I'd probably be willing to spend a ton of time analysing to find the root cause if I could. Dump the memory image, debug the code, and figure out why it's just that (and possibly other) keys. My guess just from the description is that a bitflip happened e…

Some people love the chase.

Re: Server BMCs can need to be rebooted every so often

#40

Post by aetimmes »

roughly wrote: post #9

Someone once told me that nobody ever returned a server because the BMC sucked too much, and that says everything you need to know about BMCs. Incidentally, at a previous gig, the software that handled issuing BMC commands had a BMC reboot as the first step in any script it ran.

Anecdote, but: I've seen a previous employer blackball a hardware vendor because of terrible BMC support.
Post reply on HN