"So why does the term “elite” become attached to certain languages over others?" Because the causality flows the other way. At any given point in time, there are certain languages that the elite programmers tend to flock to, because they spend more time looking out for new, good things that the average person (part of what got them to "elite" in the first place), and therefore finding someone who knows that language…
The irony, in my opinion, comes from the fact that there is no official membership mechanism when it comes to belonging to an elite. People that wield power withing a community because of their above average skill set have mannerisms, and often, as said above, agree, sometimes silently, on a certain code of conduct or modus operandi. People who want to belong to the elite start to emulate these mannerisms and identity forming mechanisms, reinforcing them over time.