Unconstrained property ownership has consequences. As a society you have to understand and decide how much you want to regulate how markets and limited resources can benefit some and harm others.
There is clear social harm to have cities filled with empty homes while there are people who want homes and can't afford to buy any on the market or are forced to commute long distances.
Using real estate to store money and other speculation does harm to people who want to participate in the market to actually use the limited fundamentally necessary resource.
It makes good sense for property owners who actively use their property themselves to have more rights than those who seek to profit from market conditions or others' use.
Having paid taxes is a weak justification for any freedom.