Every so often one of these articles comes along and someone writes at length about the struggle of living in the bay area. I've been here over 13 years now and these articles are a constant.
First, the Bay Area never promised to be anything to anyone. It had its roots in the gold rush and then the military, followed by the semiconductor rush and today the internet. At each iteration, people flooded into the region to work in lucrative jobs and push out the farmers that lived here for generations.
There may have been a larger creative scene in SF, but it never was as rich and diverse as people think they remember it to be. 13 years ago, the entire eastern 1/2 of SF was abandoned warehouses and homeless encampments. I mean, potrero hill was where you could go buy your coke, it was full of drug dealers! Now look at that neighborhood.
But ultimately, SF is a small city, not that diverse and lacks many of the things that make paying NYC prices worth it for NYers. That has been true for its entire history and I don't think that will change.
However, if creative industries are your thing, and what you crave, move to LA or NYC. That's where you'll find all of that, from amazing artists to bands that are amazing, even if they're playing at your local pub.