I will speak from my experience. When I was younger, I considered myself very smart, and I was pretty unhappy. As I grew, I met people much smarter than me; I stopped seeing myself as "the smart" one, and now I am pretty happy. It's not that I became happier because I became less smart; I became happier when I stopped thinking my value was based on my intelligence, so I stopped trying to be recognized by my intellect…
I think however your take on it is closer: when you're smart, you're told you're smart (both as a positive and as a negative - I was bullied for it), it just becomes part of your self dialogue and identity and it gets in the way.
It's taken years of therapy for me to get to the point where I realised none of that really mattered as much as having great relationships with people I liked, and going for long walks in green spaces and not thinking about anything.