Why is San Francisco public transportation so bad?

But if you're talking about the Bay Area, then you're talking about a pretty massive region composed of some pretty geographically-isolated subregions. You have the city of San Francisco, then the long peninsula, and then finally San Jose. But then wrapping up the other side of the bay you have several other cities and areas. I'm not sure that it would make sense to combine that.

This is a popular explanation of the fate of the Pacific Electric system, but like anything in history, it seems there were a number of factors that led to it happening.

The original system was designed as a loss-leader by Henry Huntington for his business venture of real estate sales and land development (e.g. electrification). Access to downtown LA from these new suburban developments in a pre-automobile world made them attractive to new buyers. As automobiles became common, three things happened:

1. The need for such a system began to decline as more people could afford their own personal transportation. This led to a decline in ridership, and hence revenue.

2. Because much of the system was at-grade (street-level), the increase in automobile traffic directly impacted service of the streetcars, as they had to share the road with the new influx of cars. This made the system less attractive, further decreasing revenue.

3. This declining revenue stream led to the system being sold to the local government, who inherited the problems above. At the same time, the freeway system was being planned (which at one point envisioned train tracks running down the middle of the freeways!), and this proposed system plus promises of more flexible (and cheaper) service via automobile busses made the system even less attractive to maintain.

So yes, while the system was eventually sold to a group with ties to the oil/auto/tire industries, the local authorities were all too happy to get rid of the system. While today the debate continues over busses (cheaper) vs trains, the real issue IMO is control of the right of ways. IIRC, MTA held on to a few ROWs and was able to use them in the new system's Expo line, among others.

EDIT: LA's MTA still faces this issue of at-grade tracks being impacted by traffic. Because it is cheaper, it is used substantially in both the Blue and Expo lines, and I've heard some arguments that the Expo line isn't much faster than driving because it has to deal with traffic+lights. From a purely financial point of view, some have argued that you get the most bang for your buck with lines like the Orange line: busses on private roadways, so you get the fleet cost of busses but the time benefits of traditional train tracks separated from traffic.

I know this could quickly turn into, "well, at least we aren't x", but Dallas has decent rail service compared to a lot of areas.

15 years ago the DART rail was just getting started. Now you can take the red line up to Plano, the blue to Rowlett, the green to Carrolton, and the orange to DFW. Since 2010 the green has been expanded to hit areas like Fair Park and Deep Ellum. You can even take the slower TRE to Fort Worth.

Is it perfect? No. But there's been pretty rapid expansion and considering how spread out Dallas is and the fiscal conservatism here, it is not too bad. Austin (albeit much smaller) does not even have a rail system and Houston's is nascent.

I'm not too knowledgeable about the situation outside the state, but I'm guessing Dallas has one of the better rail systems in the South.

The public transit is worse, but the city itself is smaller, and Uber/Lyft/etc rates are low due to competition, which effectively means personalized pickup/drop-off throughout most of the city is <$10. For an individual, that's often worth it. For a group of 2-3, when compared against the sum of their total public transit costs, it's almost always worth it.