He's not opposed to smartphones, he's opposed to
cellular phones as these can serve as a tracking beacon, following your movements.
Given that the cellular providers are capturing and archiving location data, this is fact, his conclusion is we should avoid using these sorts of phones completely. Why? The reasoning here is a awfully thin, but has something to do with "being tracked = bad" and then goes into crazy territory from there. It's the same thing with credit and debit cards. They can be tracked, therefore bad, therefore nobody should use them.
If he's concerned about remaining invisible, then this must be applied rigorously across all aspects of his life. Does he wear dazzle face-paint or glasses with bright IR LEDs on them so that CCTV cameras can't pick him up? Does he only use methods of travel that require no identification? If the FBI wanted to retrace Stallman's activity on any given day, it'd take hours at most to piece it together.
The sign that someone's a crackpot is in how inconsistent they are in applying what they've concluded. It means they're missing something important.
For example, there are people that have a genuine need for absolute secrecy, that need to remain invisible, yet they still use cellular phones, email, and social networks. They're aware of the same risks as Stallman, but they take precautions instead of avoiding them completely.
It's notable that Osama Bin Ladin was taken down because he'd gone to such great lengths to avoid being tracked that he stood out as an anomaly, an approach that proved to be self-defeating. He had this large house, but a paranoia about electronic snooping so severe that he had no internet connection, and that alone made that house highly suspicious. If you're that affluent, you have an internet connection, even if you barely use it.
Everything Stallman advocates to avoid detection just makes him an even bigger target.
