Capitalism inherently exploits economically unequal systems, and thus incentivizes them in both the private and public sectors (especially in the US where politicians are basically owned outright[1]). So, yes, it is "companies"... or more specifically, large wealth-owned profit-takers, that are the root cause of the supply-demand curves AND the zoning AND all of the other contributing factors. The fact that there are different sets of capitalists (rent and investment exploiters versus labor exploiters) who have varying influence on politicians is secondary to the fact that the politicians are primarily if not solely doing the bidding of capitalists. Attempts at corporate externalizing of risk happen in multiple contradictory directions, thus causing inconsistent public policy. Regardless, privatized-profit motivated externalization of public risk is the core problem. GDP-centric increases in productivity mean precisely fuckall for working class quality of life until we solve for the fact that the entirety of increased productivity gains have gone to the wealthy in the past 5 decades[2]. I do think as a society we should build more affordable housing in densely populated areas, but allowing capitalists to hold the reins for that would accelerate all of the problems of late stage capitalism.
Lastly, Japan has succeeded where others have failed because of a cultural difference which highly values new builds over old homes. They dispose of their homes and rebuild, as often as the US disposes of old cars and builds replacements... So political leverage by real estate investors is less of a concern there, a 30 year old home might be considered close to worthless. At the same time it creates other risk externalization problems like a colossal amount of inefficiency and waste. The only way to get a similar result of reasonable pricing in urban areas where real estate is profit-driven as opposed to use-driven, would be though affordable public housing as a counterbalance. We've seen time and again that capitalist housing development will naturally result in more capitalist-driven inequality. Investors will continue to buy making the bubble worse, then when the bubble pops the rich buy even more stock as disproportionately middle class and poor investors are forced to liquidate, as happened in the 2008 bust.
[1]: https://www.youtube.com/watch?v=5tu32CCA_Ig
[2]:https://www.vox.com/policy-and-politics/2017/8/8/16112368/pi... (or the same chart any of dozens of places)