"Building straight into the sky is easy."Except, it isn't. Construction costs of taller buildings scale non-linearly with every additional story [1]. Building skyscrapers costs upwards of $300 per square foot, whereas single-family homes cost a fifth or a sixth of that. It's so much cheaper to build low-rises, in fact, that virtually no developers will take on a high-rise project unless the property is so expensive that it can command luxury prices. This is more-or-less true for mid-rise projects as well, though at a lower price point (~$200 per square foot).
The conversation about planning and NIMBYs is misguided. As the article concludes, the city is out of land. Any new construction requires buying and knocking down something else, and the choices are all various flavors of "expensive" or "inconvenient".
For example, east of Divisadero, the density of SF is actually pretty high:
https://www.google.com/maps/@37.7907112,-122.424097,3a,75y,4...
But west of Divisadero, there are huge sections of the city that look like a suburb:
https://www.google.com/maps/@37.7468355,-122.4598049,3a,75y,...
...the problem is, everyone wants to rebuild the dense parts to be skyscrapers (which is very expensive), and transit access to the suburban parts is quite bad. Developers won't build even medium-density projects in western parts of the city, because there's no demand, because there's no transit. Instead, they build higher density in the Mission and SOMA. These buildings will always be expensive, "luxury" units, because at $300+/sq.ft. construction costs the economics don't pencil out otherwise.
[1] https://westnorth.com/2014/03/18/between-rocks-and-a-tall-pl...