I was a math + physics double major in college, plus I was also deeply into programming, electronics, and music.
In my view, the starting point for physics is a deep curiosity about how things work. But I'm not sure that aspect of it is ever taught. Rather, it's assumed that good physics students arrive at college, having developed that instinct on their own as kids -- taking things apart, breaking things, asking questions, maybe having curious parents.
Instead, the emphasis in teaching physics is almost purely on the math. Certainly, what defines physics as a unique discipline is the interest in studying problems that lend themselves to mathematical analysis. Solving the textbook problems involves identifying the equations corresponding to the wording of the problem, then solving the equations. That's a skill, but it's not really physics.
I got through my physics courses on the strength of my math skills, but was extremely fortunate to have picked up the empirical half of physics on my own, through my hobbies, and from the curiosity about nature that my parents encouraged. But if someone lacks that background, I could see them being good at math, and maybe getting a good way through school physics, but never really getting physics as an end unto itself.