Part of the problem is government regulation. In some places, it's almost impossible to build new houses due to regulations. Some portion of the population wants to see house prices continue to increase and they will vote for politicians who will implement policies which will support that agenda. Of course this happens at the expense of another, also large portion of the population who do not own a house and have to…
The demand side of rent prices is generally based on the surplus value created by an area for its residents. The value of rent prices is all about providing signals about relative utility to potential residents. This is key because it means that absolute rent is largely unimportant, the relative rent is the key. All rent paid above actual services provided does not create economic value, it instead captures surplus value from its residents and delivers it to rentiers. The issue with rent control and other similar regulation is not that it limits overall rent levels, it’s that it distorts the relative signaling. However this needs to be balanced with the negative effects large rent increases have upon tenants. A world where people are constantly forced to move due to a rapidly changing market might be price efficient, but it creates lots of negative externalities. The key here is establishing rent rules that are loose enough to create accurate price signaling, but tight enough to minimize these externalities.
The demand side for purchasing for individuals is based on individuals and companies evaluating on a range of criteria which largely boils down to the utility it directly provides, the rent you can charge others for their utility in the properly (or conversely, the rent it allows you to avoid), and its value as a speculative investment. However it is clear that in the problematic markets we are discussing, the prices are extremely dominated by the latter two.
This leads to the supply side of this also being somewhat counterintuitive. People seeking to invest their capital in property actually have two primary ways of profiting: via rent collected or by appreciation in property value. If you own property, and you have additional capital to invest, do you invest in improving your existing housing stock or into buying additional property? The answer is you invest in whichever has the highest rate of return. In low housing stock environments the answer is going to typically be to invest it in additional property over improving existing properties. You might think that rising rents would change this ratio, but this gets it backwards. Rising rents leads directly to increased property values and so largely has the opposite effect, it encourages investment in buying property over improving it.
This makes it clear that to increase investment in housing stock, your best method is to discourage speculative investment in favor of investments that raise the quantity and quality of housing stock. The way you structure regulation, incentives and taxation should reflect this. Everything else is noise, and doesn't actually solve the problem.
Side note: An exception here happens in cases where development leads to massive differential between start and end value. This is why in these sorts of high speculation environments development becomes highly slanted towards turning low density buildings into massively high density, high margin buildings. This is generally not ideal, as this type of construction has lots of negative externalities on its surrounding neighborhoods, and tends to make communities averse to further development. Making lots of small incremental improvements in housing stock is far less disruptive to neighborhoods but will only really happen in environments slanted towards improvement over speculation.