1. Values go up because of density changes related to location desirability. My house is more valuable because more people want to live here, but space is finite. This should lead to it being worthwhile to pay me to leave so an apartment complex can replace the single family homes on my block. But it's more likely that someone will pay a lot to have the single family experience at a larger price.
2. The value of location has increased. I could buy a nice home for $350k in a small town that would go for $650k in my city. But we can't move there because the downsides of working there for my SO are too large.
3. Not being adjacent to a major urban area is in many respects like going back in time some proportion to your distance from one. Finding appropriate doctors is harder. Finding employment that has modern working conditions is harder. Etc.
That said, I think it's reasonable to expect housing to be stable relative to inflation rather than being a high performance asset. And retaining wealth at the rate of inflation minus borrowing costs should beat just renting at the rate of inflation without a need for tremendous price growth. But there's obviously a big issue with supply.