To be clear, I'm not saying freedom of speech is a bad thing. On the contrary, I agree protecting speech is a great way to ensure the free exchange of ideas. (As you noted, if you can't speak your ideas then you can't freely exchange them.)
But there's a lot of "speech" that should be and is already prohibited, both legally (CP being an obvious example), or through social standards of conduct (screaming expletives at random passers by is liable to get you kicked out of just about any venue).
In my view, focusing on the goal of freedom of speech (which is to say, the free exchange of ideas) rather than on speech itself communicates much more clearly on why the principle is important and where the line is. It makes it obvious why CP is not legally protected but Nazism is. Both are despicable, but one needs to be protected in order to preserve the free exchange of ideas while the other doesn't.
And again, I'm not saying other forms of expression that don't convey ideas shouldn't be allowed; just that the reasons why they should or shouldn't be allowed are separate from the reasons why the free exchange of ideas is important.
Yes, this wording does narrow my argument somewhat. But I think it largely narrows it in ways that make it easier to defend rhetorically, especially when trying to apply the principle to non-government entities such as the large social media conglomerates that own our modern town square.