While a lot of people complain about "NIMBY" in the bay area, it's never been clear to me why I have an obligation to rearrange the area I live in to accommodate others who want to live here. People speak about a "shortage of housing", but another way of saying this is that there is a "surplus of high paying jobs". If someone can't afford to live here, then economic forces should make it more attractive to live elsewhere. Employers should see this as well, and relocate their business elsewhere. That's what happened with Intel, and many companies are now focusing their growth outside the bay area.
Just increasing housing density without solving transportation problems will result in a seriously changed lifestyle here, and that is what many people fear. Moreover, efforts to build better transportation in the bay area have been very ineffective. The light rail in Santa Clara County is a good example of this - it crawls through downtown San Jose, has very little ridership, and recovers only a small fraction of operating costs from fares. The jobs have almost always showed up in areas where there is an existing shortage of transit and housing - e.g., Sunnyvale, Cupertino, Mountain View, and Palo Alto. At this point the only feasible solution is to turn 101 and 85 into bus-only roads. No more single occupancy vehicles. Then let the market sort out how to provide transit for employers. That would be massively disruptive to existing lifestyles, but probably the only feasible way to increase transportation throughput in the face of increased housing density.
Most of the readership on hacker news is in the tech industry, but I think the biggest problem with housing in the bay area is for non-tech workers.