In my experience, the perception that "solar isn't for everybody", especially when using housing examples, is fooled by hidden assumptions that were made during the time.
Consider the situation in 1950 when you are building a house, you can double or triple the cost of home construction by insulating it so that the net energy needed to heat or cool it is minimal, or you throw a oil burner in the basement which is burning fuel oil that costs a few cents per gallon and keep everything nice and toasty. The "obvious" choice there was not to spend the money on insulation but instead to just use really cheap energy to manage the temperature range. Makes everything much easier to engineer.
If, on the other hand, you design with the assumption that energy is extremely expensive and so you minimize the need to use it to regulate temperatures within a house, you can design a house that is temperature stable with the minimal amount of energy input for air circulation.
That gets you houses in the New Mexico desert that need no air conditioning (air cooling) and churches and office buildings in the northeast that need no additional heating.
Things that I have read about include extended depth insulated exterior walls. "Smart" glass windows that reject 97% of the infrared and ultraviolet spectrum (I've got film on my house windows that are not that good but they do a tremendous job of minimizing heat load in the summer). Passive heat exchanger systems that keep the temperature balanced between upstairs and downstairs, and solar roof tiles that provide both insulation and energy for running the house.
Does that help you if you're living in a 'mid century wood frame house', probably not. But it isn't that solar couldn't meet the heating and cooling needs, it is that combined with good house engineering this is already a solved problem.