Your problem is not with SMS as a second factor though. (Unless you think the attacker had your password as well). It is with the use of SMS as a single recovery factor. The very things that make SMS a uniquely good second factor make it an awful only factor. Use of SMS for account recovery should in general (or at least for important accounts) have a delay (order of days) that allows the real user to intervene.
One option I’ve heard might be different is to not your your mobile sms on accounts, but to get a voip based sms number. It might leave things at the mercy of a different system but the footprint might be different.