High security services should send a pair of U2F keys to each and every customer when they sign up (or hit a retention/value threshold), with instructions on how to store them (that is, different buildings). Then they can use normal app-based 2FA day to day (NOT TOTP as that is phishable), and use the preenrolled U2F hardware tokens as recovery methods when the user inevitably loses their phone and needs to re-enroll…
At this point I think the thing holding back U2F is just user experience. It is not "hard" but it is a pain in the ass and most people just find it annoying. The other issue is that you ultimately need some sort of fallback mechanism if someone loses their keys. And it will happen. So you still end up with a process that can be socially engineered, which is generally the weak link in any authentication system.
Doing 2FA via app is fine for most users. The failures happen when users lose their phone and need to reset 2FA. That's where the pain in the ass (but secure pain in the ass) of U2F would come in handy, to re-enroll primary 2FA.
Nobody presently has good ways of doing 2FA resets. U2F hardware is a near-perfect solution.