Housing is actually a fairly competitive market. In a given city there are many thousands of housing units with many thousands of separate owners. Nobody has market power because there are so many other units you could take instead.
In that kind of market the long-term price is going to approach the cost of constructing new units, because if it was higher then someone could make money by doing just that, and they would, and then the increase in supply would lower the price. Supply and demand is in econ 101 because it's generally what actually happens.
This is why existing property owners don't keep prices high by colluding with each other -- cartels don't work with that many people. They have the government do it, by restricting where and how you can build housing, imposing expensive requirements on it, lowering the incentive to do it through rent control etc. Then when demand goes up through population growth, supply can't respond and instead prices go up.
If you want to claim the ordinary way that everything works isn't how this works, you have to explain why. "Not all housing units are perfectly fungible" doesn't do it -- in that case they won't all be the same price, which is obviously true, but you're still left having to explain why the cheapest ones aren't actually cheap and why nobody would want to make money by building more when prices are high unless the regulatory environment was suppressing their ability to do that.