I had a similar view as a kid, driven mostly by the fact that I wasn't very good at memorisation. I found that I could get by in maths without it, eg. deriving the quadratic formula by completing the square rather than memorising it. I didn't value memory much as a tool in maths.
After having studied maths and physics at university and worked as a programmer in a mathematical field for 20 years (and studied much more maths in my spare time), I now see my poor recall as the limiting factor in my abilities in maths. The main reason I don't think I could ever have been a professional mathematician is that I would have reached (and have reached in my own learning as an amateur) a ceiling.
I have maths books that are beloved to me, that I have read multiple times (actively, working with pen and paper as one should) and which I will enjoy again in future. But the concepts in those books do not remain in my mind. I don't reach a point where the structure of basic linear algebra, say, is baked in.
I am good at the problem solving, but maths is an edifice, one people have been building onto for millenia. I explore that edifice, and keep returning to my favourite bits of it, but the portion of that structure that is resident in my mind is, and I think always will be, small. It's a window, and as more comes into it, more slides out. Everybody must have such a window, but I know others have much larger windows than me. And that's fine - I'm a programmer, not a mathematician. But I think it's something I would have benefitted from understanding earlier in my life. Perhaps I would have set about "learning to learn" differently. Rote memorisation and active curation of memories already formed could have benefitted me greatly.