I was the person who wrote that super-long take on the San Francisco housing crisis in spring on TechCrunch.
http://www.techcrunch.com/2014/04/14/sf-housing/I'll just say this. It's four issues compounded.
1) Growth controls enacted in the 1960s and 1970s: Beginning in the late 1960s, the California Supreme Court started making a series of anti-development rulings: for example, one decision ruled that cities didn't have to compensate property owners for down-zoning their property, which reduces its value. This, of course, reduced financial risks for city governments that wanted to downzone massive tracts of land in their territory.
At the same time, cities across the Bay began enacting growth controls. Petaluma was the first in 1972, with an ordinance that restricted the number of houses that could be built per year to 500. (Economic studies comparing Petaluma with adjacent cities showed that its property prices rose at a faster pace than cities w/out growth control.)
Anti-growth measures began being widely adopted across the entire region. The behavior of California home prices started deviating from nationwide norms around 1975 or so. The problem became especially acute by the late 1970s, and so homeowners and tenants gave themselves a very sweet deal by effectively freezing their property taxes and rents via Prop. 13 and rent control. This insulated a majority of voters in cities like San Francisco from having to make tough decisions about in-fill development to match population growth. At this point, the mismatch is so out of whack and land is so valuable in San Francisco, that building a lot more housing probably wouldn't make it affordable to, say, a teacher salary level. Construction costs are $500,000 per unit, so your base price has to be at least that.
2) A byproduct of tech-dependent economic development strategy happens to be a lot of income inequality. I haven't seen a good solution to this. After manufacturing jobs dissipated from the U.S., a handful of cities like New York, Boston and San Francisco were successful at reviving themselves through a route of attracting highly-skilled, high-income workers. A consequence of this is a barbell-like jobs market where you have highly compensated workers in finance and tech on one side, and a whole cohort of service workers on the other. I've seen this wherever I travel. I went to Tel Aviv, which is arguably more dependent on tech than San Francisco with 10-11 percent of the workforce in the industry versus 6.7 percent here. Same issues with income inequality, inclusiveness and diversity.
3) Globalization of capital flows. Real estate used to be more of a local business, but now a lot of local firms have been swallowed up by REITS, which don't have any connection to the local community. At the same time, the Chinese property market is collapsing, sending flows of homebuyers and their cash over here. These are people who may not intend to actually live in the Bay Area, yet they buy up homes without even seeing them. The city doesn't know how to police this. How would you build a law or tax around regulating this behavior? New York may attempt a pied-a-terre tax, but no one really knows how to deal with this issue of vacant housing units held as investments. Technologically, it's easy to track who is doing what where at all times. But legally, there are a lot of issues around privacy with policing this if foreign buyers are taking up 1/3 of the new housing stock as investments.
Similarly in the venture world, while overall VC funds being raised is lower than what it was during the first dot-com bubble, according to the NVCA, there's a lot of PE and foreign investment that I don't think is calculated into the overall numbers.
4) Last is the lack of regional coordination. New York is 8.4 million people under one government. London is also 8.4 million people under one government. The Bay Area is 7 million people under 101 city governments and 10 bus and rail systems. There have been numerous attempts to consolidate power over the last century, but all have failed. The California state Constitution also deeply empowers its city governments compared to other states. So what happens is the rich cities don't build any housing, because they like their property values and don't like traffic congestion, or to crowd their schools. This drives prices up in the suburbs to a point where, on balance, urban environments look more attractive -- not just because of changing tastes, but also because they're not that much more expensive than the South Bay. So young people end up crowding into these poorer disinvested neighborhoods like the Mission or Oakland, which then displaces minorities and the elderly far out into the exurbs or even Central California.