Earlier quoted context omitted.
> The exact same thing that makes me think I can't just walk into your home and take what I want No, it doesn't work that way. My house is not your house. My land is not your land. If you're going to restrict my use of my property to increase your property value, you should pay me for that privilege.
No, that's not the way it works. When you move into a neighbourhood, a city, a state/province, and a country, you implicitly agree to abide by the rules. If the rules say you're not allowed to build your own house without approval of the design from an architect and an engineer and then you ignore them at your legal peril.
The rules in many areas used to forbid selling houses to black people.
Those rules were changed. By force.