I do. For some reason ssh keys became the group-think security advice to repeat ad nauseam. I often find people have only considered this very shallowly, and their reasoning is just "But OMG, entropy lolz" without actually seriously considering the available entropy and likely attack vectors and failure.
Why?
The benefits are largely theoretical if you choose sufficiently strong randomly generated passphrases, with some symbols and numbers mixed in, which I set my password manager to do. (Actually, I have a couple of super important, super long ones I keep only in my head.)
The draw-backs however are several:
1. I find that its not uncommon I find myself having to chain together ssh tunnels and other strange things to debug networking issues and need to login to another machine from a new machine that doesn't have the ssh-key. Treating servers as cattle exacerbates this issue since it's more likely to occur.
2. If a machine or server I use to manage stuff as a gateway/proxy/vpn entry to my network has all my ssh keys on it, and it becomes compromised because of some 0-day, the attacker now basically has access to... multiple entire networks. Now, this is technically true of my password manager as well, except that the context / IP addresses, etc, is not going to be as obvious as it will be on the server where the attacker can see the running services, check logs, command histories, etc. With the password-based management, sure I could get hit by a keylogger, but everything won't be compromised by default.
3. In my experience, you're more likely to lock yourself out than let an attacker in. You basically still have to have your keys managed via some type of password manager anyway because otherwise you run the risk of getting locked out of everything in case you're away from your primary machine and need to address some emergency or something.