I see your gotcha and respond by saying I'm at least
looking at other phones, but until the FCC mandates the opening of basebands, building a phone with trustable software is still a long way away.
Pine64 and Purism are making headways on that front, but again the situation is too thorny and regulations are hamstrung by the relevant companies bidding for spectrum. I've not heard of anyone having a smooth experience with PinePhone, Librem 5s, or the OnePlus line of phones.
Also, how many of these devices just run Android? So Google's got your ass there anyway. The ones that don't are foolishly following XDG and so are GNOME, KDE, or Enlightenment stans. The moment upstream changes, they'll be forced to go along because they don't fully own their UI experience.
We need portable libraries for accessing baseband stuff so that no vendor can lock a life feature (smartphone) behind their ecosystem.
I actually don't much like my phone. It's the most reliable way to find out where I am or what I'm doing, and who honestly deserves that information just because? I guess I wish I didn't use a phone at all. Despite its conveniences, on paper it's really just added a bill and a constant mental "on"ness that stresses me out.
So, not much of a gotcha because I've thought about this and it bugs me, but there's also no actually good alternative ready yet. When I'm able to BYOD with free hardware on a mainstream network and there aren't signal or reliability problems, I will. The whole point of a cell phone is remote availability, so it needs access to a reliable network to be of any use.
Please, if my understanding of free phones is out of date, correct me. It'd be nice to get one step closer to a completely user-controlled cellular device.