For context: I moved from NYC to Philadelphia, to finish my thesis and be with my wife who works at the world's best children's hospital. I have lived in Portland, Chicago, and spent extensive time in Toronto. I haven't spent much time in SF.
The change in cost of living is dramatic, and the quality of life is higher by far. I have eaten in extraordinary restaurants in my life and I don't feel Philadelphia suffers in that regard. There is a similar caliber of restaurant (with the exception of a very select few you are likely to eat at a few times a year and could honestly fly to and still save money). While I'm sure ballet and music are better in NYC, I'm not sure most consumers can tell the difference or whether it's worth living nearby.
I don't like remote work as much, and I make no arguments about remote work vs onsite. The talent pool in big areas is probably a huge plus for expensive areas (although healthcare related industry is probably great in Philadelphia, so maybe I haven't lost much there). And if you need onsite workers (again, that's maybe best, I have no argument either way), you can just hire them in SF. (You could probably pay 2x going rate adjusted for cost of living in Philadelphia and still come out ahead, but it might be harder to hire replacements). Quantity of cool jobs is definitely lower.
But if your goal is just financial independence, you could cut your rent to 20-40% of NYC and be within walking distance to work here, it's like getting a 3x pay raise.
I miss certain things about NYC, to be clear. Philadelphia is smaller, there's less of the intense artist feel, less craziness in the streets and much, much more typically dressed men and women. I miss hundreds (literally) takeout places that deliver amazing food in 30 minutes.
I do not miss the trash in the streets (piled three times a week in big bags in Manhattan ) the smell (everyone knows that returning-to-NYC smell), the rats, or the 500 sq foot apartment I was lucky to get in Williamsburg that pales in comparison to the apartment I leased in one of the top ten neighborhoods to live within North America here in Philadelphia.