Speaking personally, I feel like my primary concerns are:
- ease of use (if something is a pain in the ass, I've found I won't use it in practice, in spite of what I know is in my best interest)
- credential stuffing/data breaches
- phishing
- getting locked out of my accounts due to losing my credentials
In particular, I think that, although a password manager plus any sort of 2FA mitigates credential stuffing concerns pretty well, HW 2FA is probably more effective against phishing compared to a TOTP app or email 2FA[1]. Purchasing a single backup key that I have a trusted family member hold on to, plus printing out recovery codes and storing them in the same place I keep other important personal documents (passport, birth certificate), is for me is a pretty reasonable hedge against lost credentials[2]. I also personally find HW 2FA to be much more convenient than TOTP authenticator apps in general[3]. Thus, I use a hardware 2FA token (at least for "important" accounts) and sleep pretty soundly at night, but that's very much the result of my own specific concerns, preferences, and priorities.
[1] A hardware key is able to validate the domain directly and mainly relies on the browser not fucking up[1a], whereas SMS/email/TOTP tokens require the user to evaluate the authenticity of the form they're filling out. Obviously there's some subtlety here, since a PW manager browser extension can also do domain validation, but at the end of the day, I think there's some non-trivial benefit to using a HW token.
[1a] Chrome's WebUSB API is an interesting case study here.
[2] I actually lost my primary HW token earlier this year, and so got the opportunity to test out my recovery plan. Having done so, I'd recommend anyone who depends on HW tokens to go through a similar exercise (preferably simulated, rather than "for real"), if nothing else than to get some very satisfying peace of mind.
[3] I suspect this is not a universally held opinion.