Why the foundations of physics have not progressed for 40 years

I 100% agree with you in all respects — I don't come from a physics background but I hear you loud and clear. I think the 'why' is deep and psycho-historical in nature:

- we're exiting the "industrial" mindset where everyone is the same making the same products, to a wider topology of knowledge and skills (more and wider horizontals, more and bigger verticals, 'average' profiles become 'scattered'). This clearly drives a need to "learn a little bit of a lot of things" even at expert level.

- The walls and denial you expose here is to me but a symptom of the disease that current academia will either have to heal or die of. Seeing how Khan (and thousands of Udemy's after them, indies) changed the landscape, my money is on a major paradigm shift incoming for academia (it's already done, they just don't seem to know it yet as institutions, most of them). Lots and lots of great teachers around the world almost freely sharing incredible hands-on knowledge and insight.

- Some applied domains with dramatic tension of the demand side (lots of positions to fill) don't have the luxury of elitism and massively adopt "pragmatic" approaches especially in learning. Software dev, programming and tech in general is much like that — the "one liner" installs and 1-page "getting started", all the intelligence solely put into making things intelligible and usable is, frankly, quite humbling and inspiring in that field. A very good side of the SV/Cali culture. So, examples of how to proceed next really do exist.

Now when I think back of topics that I hurt my head against for months or years, that a simple 20-minute video could 'unlock'... Why, why do we not make it a staple of "teaching" to at least consider 2-3 angles to make sure everyone's got a fair chance at getting at least 1?

- On the topic of hubris and laziness, this is where physics went astray, imho. Too much hubris and not enough laziness. That was back in the 1980s and it took 40 years to realize, probably 10-20 more to "fix", if ever before we build a new system (see above).

That being said,

> Geometric Algebra (GA) was my "lightbulb" moment where I finally understood where Dirac matrices, Pauli matrices, and the like come from and why they have the structure that they do.

YES, please! Geometric algebra seems like the thing that could blow my mind too. I am very visual, to a fault maybe.

Would you have a 'favorite' resource to share? (book, course, youtube, whatever?)

>> What avenues?

Pilot wave theory is one. The other was a paper - forgive this explanation - the found equivalence between particle physics and fluid dynamics. Suggesting the objects in physics might be modeled as say vorticies in some kind of fluid (aether). The equivalence was IIRC only to first order, but the work to get there must have been a lot. I'm sure there are others.

That’s like saying, “Cat is a word used in a language to represent a physical thing.”

> 1. I originally thought the Heisenberg Uncertainty principle was a natural consequence of using particles (photos) for measurement. Instead however it seems to be a fundamental property of the universe, which I only learned after finding out most of the mass of hadrons comes from the relativistic motion of quarks and it explains why hadrons don't collapse to a point.

> 2. What does it even mean to create more space? The universe is expanding. Ok, I can accept that. But what does it mean?

IME both QM and Relativity are much more easily understood if you start from the actual equations and/or an undergrad textbook. I don't think physics is - yet - beyond the technical and scientific minded amateur who is willing to read and understand some equations. But it's probably gone beyond the ability of any science journalist to render into prose.

> 3. I find the models for dark matter and dark energy to be... unsatisfying. I realize there's experimental evidence for unobservable mass but it feels like a fudge.

I think working scientists would agree with you at least as far as dark energy goes. Dark matter is pretty indisputably just some kind of matter that we can't see (see e.g. the Bullet Cluster) - there's still something to be solved in terms of figuring out what it actually is, but I don't expect that to be major new physics. But as to dark energy: yeah, it's a fudge. Everyone knows it's a fudge. But it's still our best description of reality. Even if you knew there was something wrong with epicycles, they were still the best way to calculate planetary orbits at the time.

> I could go on. I don't for a second mean to suggest any of these notions are wrong. It's just that the models have gotten so complex (it seems?) that it just feels like something huge is missing, something that will eventually seem obvious in hindsight. Or am I just a lemur trying to figure out how an airplane works?

I think models aren't complex so much as unfamiliar. We're getting further and further away from everyday experience, and so more and more of the fundamentals of reality have to be understood from mathematical first principles rather than everyday intuition.

The models are actually really good though. Certainly once I understood QM it seemed so clear and simple that it couldn't possibly not be true.

First off, mathematics did actually reinvent itself in the late 1940s, with the discovery of Category Theory, which, despite being sometimes called “abstract nonsense”, has found its way into theoretical physics. Further, there is nothing mystical at all in any of the concepts of the modern physics - not more anyway than in the concept of, say, the atom. It is actually math that should be credited with the removal of the shroud of mysticism perceived by some - perhaps even many - of the uninitiated. (Physics is not an exception here - some people still look at a working computer as a miracle, for example.) And the homogeneity at the scales at which there’s just too many things to allow for much diversity should, too, be seen as one of the manifestation of the absence of any true mystery in the universe.