I have the opposite regrets.
I have a very persuadable personality, and despite my anxiety in social interactions, I was often pushed into things that everyone was doing, but I wasn't enjoying doing. So much time was spent playing games I don't like, doing sports that weren't my strength.
When I reflect on my younger years, I'd probably had been less sullen and depressed if I allowed myself to stay in my apartments, studying, making music, and just not concerning so much about people's invitations. I should've just taken the damn violin classes instead of going to tennis courts; or written my damn novel, instead of going to the party watching people drink alcohol and blow on ping-pong balls.
For those who enjoy parties, they were fond memories. For me, they only exacerbated my loneliness.
There are two realizations that came to me too late: 1). I don't need to appease to every interests of my immediate circle to maintain friendships; it's in fact somewhat counter-productive. 2). however niche a hobby is, there are probably enough people also doing it to give me enough society, especially in the age of internet.
By first asserting to myself my actual passions, and seeking friends based on that, I actually get a broader circle of friends in a more organic way.