Earlier quoted context omitted.
On the other hand, it's much easier to reroute a cell phone number through social engineering than to steal TOTP secrets.
Is it really possible from an evil standpoint to get SMS rerouted to another number? I was looking into that a while ago (I wanted a prettier number, but didn't want to lose things associated with the old one) and the answer I got was that it's not something anybody can do. I get how the phone call rerouting stuff would go down, but not SMS rerouting.
Not sure if any of that still works with GVoice, but if not, I'd look into doing it with Twilio possibly.