Phone numbers are useful for exactly the reasons you find them frustrating: stability -- as you said, everyone and everything you associate with can and will store and contact you via your phone number indefinitely -- and portability -- everyone accepts and understands phone numbers, modulo international dialing.
Sure, something could and maybe should replace phone numbers, as the system is definitely messy wrt international dialing and countries changing numbering plans.. But the thing that replaces phone numbers in their usefulness will bring the same frustrations you express.
Email has mostly the same characteristics, especially for non-computer-people. My parents were paying $10/mo for dialup up to ten years after switching to DSL and Gmail, just to keep their old email address. I bring that up not to point out the extortion -- email could theoretically have had address providers decoupled from hosting provider through DNS, if it had been made user-friendly -- but to point out the value in the stable identifier. I know this is an anecdote, but the story of AOL email is similar, that 2.5 million people [1] were still paying $20/mo for their dialup and bundled email when "some of whom" (sorry there's no better information on this) had since switched to a different ISP, but kept paying AOL to keep the email.
> I even had to send them a government issued photo ID recently so I could keep the number.
Governments will always want to link users to their stable identifiers. It's in their policing interest, for better or worse. Switching away from phone numbers will just shift the problem.
[1] https://consumerist.com/2013/08/08/believe-it-or-not-2-58-mi...