I lost my key a couple of weeks ago and was surprised how easy it was to get back into my accounts with just my phone. There is no point in using something like that if providers allow you to failover to more conventional authentication methods without any hassle; the keys are useless. They are not going to add manual verification for a couple of people who lost their YubiKey.
YubiKey is useful for instances when you want to grant somebody access to something with just a key. I don't see it going beyond that anytime soon.