Math was always extremely easy for me growing up. Up through my first differential equations class I found almost everything trivial to learn (the one exception is that I always found proving things difficult).
I made the mistake of minoring in math and that slowly killed my enjoyment of it. Once I got to differential geometry and advanced matrix theory it all just became too abstract and I just wanted to get away from it.
For several years after college I would routinely pull my advanced calculus text out and do problems "for fun". After a while I stopped doing that. Within a few years of no longer being exposed to math, I found it all incredibly foreign and challenging, to the point where I would say I have a bit of an aversion/phobia to it.
I'm trying to reverse that now by tackling a topic I'm interested in but have previously avoided due to the math-heavy nature of it - type theory.
Hopefully I can find the joy in math again through this.
I think my point is that you can lose competence in math very very quickly through lack of constant exposure.
The same is probably true of programming but I hope to never end up in that position.