Server BMCs can need to be rebooted every so often

The operative word being "proper". In practice I see it accessible on the LAN far too often. The ipmi v2 protocol is so bad that if you just request to login with a known account name (which is probably 'admin') the BMC server will _provide you the password hash_ for you to crack at your leisure.

Yes, and as essentially a "hidden flat tax" it burns a lot more for large quantities of lower end hardware vs single bigger iron. Looking back at the records, at the time adding their iLO/M.2 comm card (needed to have a dedicated iLO port on those servers) was $65 from Provantage and then the license to actually do basic stuff with it like a web console was and $227. So ~$290 all told extra. We've got a few higher end systems, $10-20k NAS or larger hypervisor systems, and at that level sure an extra $300 stings a bit but is against a lot of other stuff. At medium range, more like $3k-5k, obviously worse. But some use cases called for say 6x $500-600 systems instead of a single $3k-6k system, and for a few projects someone was trying to go as cheap as possible and grabbed some still pretty new (gen10) plain vanilla ones off of used/bankruptcy sales for like $300-400 because the price seemed so attractive. But then using iLO properly could nearly double the price, and at the lower end there is a big difference in the hardware you can get for $600-700 vs $300-400, and then multiply that per unit.

Meanwhile competitor systems all have dedicated ports out of box. SuperMicro does have a paid "full unlock" for their BMC, but the only thing it does is add bios updates and such. All the core normal management functionality is there by default. It also only costs $30.

Granted HPs had other irritations like really wonky proprietary fan control (heck, proprietary fan cables too) that would mysteriously fail to function with different flavors of the same OS, and couldn't be overridden from the BMC (then what's the point!?). Also they were slow with EPYC options when that's what a lot of us really wanted to be switching for, the performance and value propositions were getting really good vs Intel who were also jerks.

Like lots of big players the experience may get different if you're buying hundreds to thousands or more units and have a dedicated account manager who takes care of all this for you etc etc. But x86-based servers is a pretty damn competitive market and at some point one has to stop and ask why hours are being burned futzing with stuff when literally the entire basic point of getting "server class" hardware with remote management functionality is to save man hours by NOT having to futz. So yeah there's a little rant I didn't even know I still had in me years later :). HP you silly goofs.

Not just BMCs...'modern' (I learned this 6-7 years ago) network cards are pretty much seperate computers that handle dataflow up and down it's stack.

We had IDSes that were happy, up, network interface counts were climbing, but they were STONE DEAF to the network traffic we were actually interested in.

We had an interesting incident where one of our datacenter temperature sensors kept on rising, and operators on site could clearly hear the noise increase of all servers going to max fan speed following a BMC "crash" triggered by a network loop on our IPMI lan.

It took us a while to identify the issue and all systems were running fine, but we had to shut down many racks to avoid the temperature to rise too high.

I've personally been involved in decisions like that, as well. If you're throwing stuff into a datacenter and the BMC doesn't work right[1], the hardware is basically a brick. Vendors should be blackballed for their incompetence.

[1]: all things i have personal experience with:

- chassis bootdev pxe => doesn't do that, just reboots to normal OS

- chassis power off => doesn't do it (oh, here's an ipmi raw command you can use for this BMC version. NO!)

- dhcp server sends an option in an offer it doesn't understand, drops the offer, no IP at all.

- gets scanned by auditor-checkbox-as-a-service (qualys), locks up and sends the host CPU into 100% and locks up that up too.

- Not supporting IPv6 properly (if there's a place to start deploying IPv6 it's BMCs), i.e. uses SLAAC properly, but doesn't use the gateway from the RA so you actually can't use it from outside it's own segment - needs a firmware update to fix but uhh, we didn't dual-stack the BMC network because the whole point is to get that IP space back.

We had to write a test suite for vendors to run against their BMC and validate these things and you were disqualified if you failed.