>> At least in my mind, the defining attribute of an HSM is protection against physical compromise of the device.
FIPS definitions help a bit here.
A hardware FIPS 140 Level 2 cryptographic module (HSM) is only required to show evidence of tampering. A $5 broken seal if often the solution. Note that applying a seal to the module in the link is not sufficient because I can compromise the thing without opening it (just press the button and reflash with non-signed images).
Level 3 is a step up in that you must have some degree of tamper resistance, but in several cases, most of this security behemoths just drop a layer of epoxy on the board and call it a day. Then they sell it to you for $10.000.
With Level 4 things get serious, but then you can count your suppliers in on hand and pay accordingly (and Safenet is not one of them).
I agree that smartcard-based solution would be ideal, but specs and native firmware and SDK are never open, so you cannot build an open-source product with them.