The money going into SF is big enough these days(just look at the city budget, it's a multiple of similarly large cities elsewhere in the US) that the government has fallen into a default of over-regulating everything and then carving out shortcuts and excemptions, and this isn't a new trend, even - SF has always been expensive but the pay-to-play system really started picking up steam with Willie Brown's mayorship back in the 90's, amid the original dot-com boom.
When a city is "hot" and the working population is very transient, which is the case with SF's new tech workforce, nobody is really being a watchdog. The existing property owners will drift along, yelling about anything that might concievably affect their property values, and they get thrown bones to stay complacent since they're the most consistent voters. But they don't pay rent, and the majority of the workers still expect to be here as a career stepping stone and then move on to somewhere that doesn't suck, and so they put up with the rent and don't put down roots. People staying long-term under rent control hang onto their position for as long as they can. There's a lot of kicking cans down roads going on, hence the system trundles on.
