Equities don't keep growing indefinitely without underlying economic growth, and GDP represents that economic growth. There is no immutable law that SPX needs to keep growing 10%.
It has been outpacing GDP because increasingly, production is being concentrated in fewer, larger companies. So right now, they are growing. There are limits to that growth if GDP isn't growing as well - there are only so many smaller companies you can put out of business.
This - investing into SPX, or VFIAX, or any other market aggregate - is a viable strategy in the face of unlimited growth. It's a reasonable mid-term strategy. It's entirely unclear that there will be unchecked economic growth for 40 years. In fact, there's very good reason to assume there won't.
As for "get some portion to work higher yields" - those yields carry risk. They're not just making you more money, you stand a better chance to lose money, too. Which in turn means much less return, even if you manage to maintain 4%.
And to make matters worse, you're timing dependent - if you'd invested $1M in 1999, you would've been back to $1M in 2012. Except, if you'd lived off it for those 13 years, you'd be down to half your initial investment, ~$480K.
FIRE is a high-risk strategy. I'm not saying it can't work out, but over 40 years, there are a lot of things that can happen, and most FIRE proponents have very small safety margins. The crux is that that doesn't seem particularly problematic in your 20s and 30s - worst case, you start again, right? Except "start again" in your 50s, 60s, 70s is a really sucky proposition.
Ultimately, everybody takes their own risks, but let's not underplay those risks.