Anyone with private key access has control over my user account and has better access than what my private key + passphrase would provide.
I understand layers (probably moreso than most), but this is something that always bothers me a lot from a practicality perspective. My passwords are encrypted at rest via encrypted filesystems. If you are running things on my personal machine as my user account, I'm already being keylogged and/or am executing arbitrary code for you. If I'm logged into somewhere via ssh (hint: I am whenever I have a network connection), you can just scan my ssh config and use my ssh key anyway. From there, you can probably do a lot of other nasty stuff. ssh-agent won't really prevent this. It will prevent the malware from working again when I reboot until I log into another remote host (which I've established I do a lot) where the keylogger now gets me.
It's possible, but extremely unlikely that I have might have completely read-only media. I could be using my TPM device to protect from from booting and executing modified system states. Some of this might prevent you from easily persisting the keylogger threat across reboots. I might also have a module or something that calculate checksums on startup of critical things, have ridiculous anti-exfil outgoing connection policies, etc that prevent all but the most targeted attacks.
I don't have all of that in place. (In particular, to anyone generating a profile for me, I don't build detailed outgoing packet filter rules (you are welcome).) But what I do have in place will probably prevent me from getting my initial passphase keylogged if I used ssh-agent since it's likely (although this isn't strictly necessary) that I'm going to get attacked again after I start logging into remote hosts. So they can't steal my password, but they do have unrestricted access to my user account and the remote users I can log into. That complicates things, but is still a major security failure, to the point where them having the passphrase to my key isn't super important. I mean, in this scenario, they already have the absolute best input vector (a history of me logging in so they can execute attacks at the times I'm supposed to be logging in, as well as direct access to the systems from my ip addresses) to the point where using my ssh key from elsewhere is probably a worse a idea.