Loneliness, learning how to exert soft/passive power and patience. As you move further up leadership, your social circle of peers necessarily and naturally becomes smaller. For those of us who get a lot out of the social aspects of work (the team sport as you said), this can make something that you'd otherwise celebrate rather bittersweet.
Exerting soft/passive power has been the major long-term challenge, I think. Learning how to exert action at a distance requires a lot of finesse, a lot of listening, and more scanning than printing, so to speak. I'll be the first to admit that I lacked some of that when I began; still do now, really. You have to think about things as an analyst, an evangelist and an investor, rather than a do-er. There's a lot of education, persuasion, and sermonizing involved, even if it's primarily in recruiting others to work your ideas or to join your team to work on theirs. Leadership is about people; people can be messy and fickle, so leadership challenges involve things that can be messy and fickle.
Finally, patience has been the most subtly challenging part of all of the above. When you're more alone and need to exert soft power to be effective, you need patience in spades, and you very likely may lack it. I certainly do. The impatience that was such a virtue to me as an IC has bitten me as a leader. I've had to learn the hard way that patience regarding challenges isn't a nice to have -- it's a necessity, and without it, I'll fail.
That said, I think I have enjoyed a lot of tailwinds that have made much of my early leadership challenges a non-issue. I'm pretty self-aware and am more motivated by winning than being right, so I've not had too many challenges with stupid unforced errors on my end. I like talking to people and enjoy fixing their problems to make them happy, so management and leadership has large component of real fulfillment to it that I don't have to fake just because it takes my career forward. I've spent a lot of time in necessarily cross-functional technical roles at startups in my early career, so it's never been hard for me to empathize with leaders in other divisions and be an asset in effectively translating technology into their business goals.
But even with all of those tailwinds, leadership is /hard/. Beyond the day to day challenge of increased responsibility, decreased agency, and thinner safety nets, it's just plain mentally tricky in so many ways I could conceive of but not fully appreciate until I began my own journey. The highs are high, but the lows are really, really low, painful and traumatic even. I never questioned whether I should be doing what I was doing as an IC. But as a leader, that questioning became a lot more prominent. And it had very little to do with how well I was objectively doing according to the CEO (which was "very well"). I think it is just the nature of the beast.