Several sites that I use, even some that support TOTP, still require a phone number. I hope that since you have to “unlock” your Google Voice number before it can be ported, it’s immune to theft (assuming you protect your GV account of course) and relatively safe to use. https://support.google.com/voice/answer/1065667#xferout
I see this sentiment posted here a lot and I'm here to say that it is sadly not a good hope. When you do a number port, the automated/efficient/normal path is that your new provider (the "gaining" provider) submits a request to the number portability authority requesting that your number be moved to its routing away from your old provider (the "losing" provider). The authority passes along the information that the gaining provider submitted and gives it to the losing provider. The losing provider is then responsible for returning an automated "yes," "no," or "wait." If "wait," the losing provider is supposed to reply again within 1 to 7 days indicating actual yes or no; if no reply, then the port will complete with no further action. If yes, then the port will complete. If no, then the port is rejected.
Now, here's the major hole: It is entirely possible to do what's called a "force port," wherein the gaining provider attests to the number portability authority that the gaining provider Really For Sure Totally Does have authorization from you (the subscriber) to take routing for the requested number. This is only supposed to be used in the case of a recalcitrant losing provider or where the losing provider has no automated system and the subscriber wants/needs the number moved Very Fast Now. But, realistically, this very much can be abused and, if an attacker is motivated, will be abused.
There's nothing Google (or, more accurately, its underlying carrier, Bandwidth.com in most cases) can do to stop a force port. All the "unlock" feature on Google Voice does is cause an automated port request to be approved if the other subscriber information matches. If an unlock is not done, then Google Voice will simply return "nope" on all port requests. But a force port can still go around that and, disturbingly, the losing carrier may not even know that a force port was done until days later when it notices that the LRN (local routing number) database no longer points the lost number at its service.
So, SMS is still a terrible idea for verification even on Google Voice numbers.