The basic problem is simple: lots of demand chasing little supply, as discussed extensively here: http://www.slate.com/articles/business/moneybox/2012/05/face... (HN discussion: http://www.slate.com/articles/business/moneybox/2012/05/face... ). See also this: http://www.theatlantic.com/magazine/archive/2007/11/a-tale-o... . This is one of these problems that has a simple, obvious solution—raise or remove height limit…
As far as I'm concerned the lack of cheap housing in San Francisco is no more of a problem than the lack of cheap housing in Beverley Hills.