I think one of the keys to understanding math, and to a lesser but still significant extent, physics history is to remember what you believed as a child, and how you struggled with the concepts taught to you. And that's even with a math curriculum designed to lead you to modern math. (One can debate how
effective it is at that, but that's a separate topic.) Those misconceptions we had as children are pretty fundamental to the human wetware. Even today, with centuries of refinement and educational advancement, really only a small fraction of people come away from school with the ability to think truly mathematically.
For instance, even "just" negative numbers is a fairly counterintuitive concept. Despite how they may seem universal today, they actually didn't pop up in all that many cultures historically before their current line. And that story gets repeated over and over for all sorts of developments. It takes time for fields of study to process and abstract these things, because they weren't just handed it on a silver platter in school.
(To the extent that that doesn't seem to be the case today, I'd say that as we have become more and more mathematically sophisticated and the area of mathematical inquiry exponentially increases, the dominant factor 'holding back' math today is our inability to cover territory. Today nobody can completely cover a major discipline before the next generation is already coming in with fresh brains. That's a relatively recent development.)