> So whenever people speak of more freedom, the question is, more freedom for whom and at the expense of whom.
This only applies at the very boundary of freedom. I would argue we are not frequently at that boundary - often freedom is curtailed for reasons other than preserving the freedom of others.
A silly example: Suppose the government outlawed wearing red shirts. Regaining that freedom would not impede the freedom of others in any way.
A real life example: It is illegal for me to buy raw milk from my local farmer. Allowing two consenting adults to make a transaction would not affect anyone else's freedom.
You can view laws on a spectrum from "strictly exists to protect other's freedoms" on the left to "strictly exists to curtail individual freedom" on the right. I would argue that making raw milk illegal is a law on the far right side of that spectrum. It is up for debate where current political issues fall on that spectrum. Gun control advocates say that the existence of easy access to guns restricts their freedoms, and so put gun control laws on the left side of the spectrum. Gun rights advocates disagree, and put gun control on the right side of the spectrum.
Regardless, nobody would argue that all current laws are at the far left. If we wanted to maximize freedom as a society, we have some easy gains before we have to start worrying balancing the conflicting freedoms of others. The problem is that most people don't want to maximize freedom - they want just enough freedom to do what they want to do, but enough regulation to stop others from doing things they don't like.