My point was that you had missed a key component: compartmentalization.
To use something that's more in the middle: you'd want to use a password manager per client (for sure) and probably divide low-security and high-security credentials.
It's just that the remote SSH servers at my job are fundamentally an expediency meant for emergency situations, and often can only be used once without being reset, eg, locking the account after a single session and requiring someone from inside the network to unlock it (after setting a new, one-time password).
In this sense, their security is primarily that I must compartmentalize each emergency server, and only use it during the appropriate emergency. However, during emergencies, the nature of the emergencies overrides the risk of compromise when accessing those commands, which typically are just specific scripts run from a locked down account.
Thus the security of the system depends on my ability to keep emergency credentials compartmentalized when using a different set, so that way only a single set of emergency commands are available to an attacker at a time -- and only when they're commands related to an already failed system.
Thus the security of the system is best preserved by strategies which highly compartmentalize the credentials, since they're meant to be used on potentially unsafe systems only during emergencies.
I guess I just remember 3-4 passwords like it's my job to -- because it is.