I think I'd like a version of this scheme which works a bit like email-based verification: you have a trusted provider (like an email provider today, you can self-host or use any of the many 3rd party hosts) that you use to vouch for your identity. When you want to log into a website you use a certificate authenticating you, the website checks if it's valid with your authority (similar to the current email-based verification on most websites). If you want to change your certificate for any reason you only have to do it with your authority and everything else keeps working as usual. The drawback of course is that you have a single point of failure, if the authority is compromised you're naked in the wild.
IIRC OpenID worked like that but unfortunately it never gained traction. It's a shame really.
More practically I do use a yubikey myself but mostly as a GnuPG smartcard, not for 2FA. I actually have the same key stored on multiple tokens as a backup, so if my current key breaks I just have to fetch an other one. Of course if instead I lose it or it's stolen I'll probably have to generate new keys (even though the PIN should theoretically still protect me) so the problem still exists.