This is why I use SMS as my second factor for my Google account. Much harder to lose. It could be vulnerable to sim swapping attacks, but I consider Google locking me out of my own account a more likely threat (and frankly I'm probably not a high-profile enough target for anyone to bother with that, and in any case they'd still need my password).
Sure, it's not the most secure way but I trust this over carriers securing my number.