Show HN: How to steal SSH Key Passphrases

I've always wondered about this, but never had the time to actually investigate if this is possible. What's the best way to protect oneself against this sort of "attack"?