We use Yubikeys as GPG smartcards, and use them for gpg-agent as ssh keys. Everyone puts their hsm keys on their github account (and removes all others). We fetch the keys for each user from github on system init. e.g. https://github.com/sneak.keys When we need to add/remove people, we just update the list of usernames in the script that fetches keys, and then kill off instances one at a time to force a redeploy.
At least github supports two factor auth, but you're still placing trust in a third party.