I've done lots of activies, including sports and dance, and some activities simply have a higher barrier than others.
Partner dancing - at least leading - is definitely on the higher side :-)
There are several factors. Leaders must endure months where they can do little or nothing, and they have to do it among people who always look great - if they social dance; if they don't, their learning time increases.
Then there's a lot of pressure of performing with and for somebody who's usually of the opposite sex (this applies also to follows).
It takes a lot (internally) before feeling comfortable enough to "simply enjoy".
Playing an instrument or practicing an individual sport doesn't have such pressures.