I use different keypairs for different trust domains. (also makes it easier to hand over access when you quit a job.) SSH keys are actually pretty horrible from a security perspective; no expiration, generally held in software, etc. And without a lot of work, single-factor. I love the ssh security model of being pretty good and better than telnet for everything (which it ~fully displaced, unlike https vs. http), but…
Wouldn't a passphrase be a second factor?
By the way, we use a security fob at work for that. Seems to work fairly well. The private key never leaves the fob, you have to press a button to sign anything, and every once in a while you have to enter your passphrase.