Thesis: If the bay area had a viable transit structure, the current housing crisis would be less problematic.
People either need to work near their jobs or be able to commute to them. Housing density is low in most places, so "working-class" people get pushed out of "nice" areas, so they need to commute. Commuting is bad because transit is bad: there isn't an interconnected, pervasive mesh of transit with capillaries reaching into neighborhoods and arteries connecting cities. We've got Bart and Caltrain, but they're each for the most part single corridors, and the local transit doesn't link up well with it. So, taking public transit costs lots of time and money. The alternative that people have is to drive, but there are too many cars on the road, so driving takes lots of time and money. The result is that people feel like they don't have anywhere to go.
I'm not sympathetic to the argument "I've lived here for NN years so I should be allowed to stay living here as long as I want, realities of economics aside".
I wonder if it would help if there were a governing body between county and state that comprised "the bay area". Individual cities and counties have to strike deals between each other to link transit, with all the usual crappiness that politics entails. If a few counties (or cities that aren't in the same county) have a disagreement, do they have any place to air their arguments aside from the state level? People in LA don't care about the bay area's transit problems. People outside the major metro areas (central valley, far-northern california) don't care about the bay area's transit problems.
Part of the problem is that low density begets low density: all those cars need a place to be parked all day while their drivers work, so we have ~40-50% of the space in cities devoted to roads and parking. Further, low density makes mass transit economically infeasible.
I don't have a solution to offer. I can't see a way from where we are to the bay area I dream of, given the state of the economy and the politics involved.
Conclusion: we should collapse the bay area into a point mass. Then everybody gets to live in the nicest digs in town!