Your problem is not with SMS as a second factor though. (Unless you think the attacker had your password as well). It is with the use of SMS as a single recovery factor. The very things that make SMS a uniquely good second factor make it an awful only factor. Use of SMS for account recovery should in general (or at least for important accounts) have a delay (order of days) that allows the real user to intervene.
FWIW I wouldn't regard SMS as a good 2nd authentication factor either, for the same reasons as this issue, it's too easy to get a carrier to transfer a number to an attacker. Where it's used as a second factor, this still has an impact which is, if an attacker can get the password (and there's been enough breaches and keystroke logging for that to be common) they can then grab the number to get full control of the ac…
But how many hardware tokens or TOTP tokens are users willing to deal with? I currently have eight for various clients and systems at work. If each online account required a TOTP token or a custom hardware token it would be a confusing mess of tokens.
I don't know if there's a safe and easy way of reusing the same token across sites. Until then SMS really is the only "solution".