Earlier quoted context omitted.
TOTP is phishable, which is a way way way more common attack than sim swaps.
Sure, no security measure is perfect. Hardware tokens are likely to have better properties than TOTP, which has better properties than SMS, which has better properties than nothing. you can phish SMS exactly the same way you can phish TOTP, I'd say :)
It also comes with large downsides. Security is an economics game. Marginal improvements in security posture are not always worth the cost.
There are a bunch of people who insist that web services should drop SMS completely and demand that all users use TOTP (at least). I question the value of this change given that TOTP only protects you in comparatively rare cases.