This was definitely the case in my situation. I grew up gifted AND overweight/obese at an athletics-focused public school in a Northeast US suburb. I was bullied. A lot of my thought processes were "it doesn't matter if I'm fat/how I look/how my peers treat me because at least I'm smart and someday I'll be better than them..."
So I spent my high school inside WoW, where nobody can see you, and I went to a good college and worked really hard to get an education, lost some weight there (and developed socially, somewhat), and now I'm a full-time software developer (graduated school last May). I'm realizing that (a) I'm not actually an "introvert," I just never had a chance to develop socially, and (b) the reasons that initially attracted me to software development (specifically, the ability to just "get in the zone" and escape from reality for hours and hours) are now considerably less appealing to me. I'd much rather be in a job that exposes me to some sort of social interaction beyond my manager, although perhaps the grass is always greener...
Anyway, to relate back to OP, I really, really wish someone had pulled me out of my public school and let me develop with a group of other "misfits." I eventually fell into that social group anyway, in high school (mostly artists, turns out), but the damage was done. I had a "challenge math" class in elementary school that was fun (and all the other kids I remember from that class either went to ivy leagues or ended up in software or both), but beyond accelerating our math education and offering AP courses, there was no support for giftedness in my school system.