Free software is about freedom. Restricting it from anyone means it's not free. There is no requirement that we must create free software but if it's called free I think it should always have the basic qualities of freedom; not only when it fits our purposes and our values.
It's a choice for the authors to make based on what type of free they believe in. I think free under MIT and GPL are two different philosophies on how you see "free". MIT: free for anyone, do whatever you want GPL: free if you also make your software free AGPL: GPL but SaaS can't circumvent the requirement to make your software free I see why principled open source proponents would select GPL or AGPL. They don't just…
FWIW I have the same quarrel with people who talk about a country being "free". To my mind, a truly free country would have no laws. It would be a horrible place, because the restrictions that laws place on us tend to make things better for everyone (we may disagree on this law or that law, but some laws, like "Don't kill someone without a very good reason", would have >99% popular support anywhere in the world).
"More free" does not necessarily imply "better"; it could be better or worse. I'd like to shift usage of the words "free" and "freedom" in this direction, but think it's probably a lost cause as the words are too emotionally charged with connotations of "good".