There are a lot of legitimate criticisms to be made of YC and of VCs in general (please do not take this as me going "there is nothing wrong in the VC world at all", because I absolutely do not believe that!), but I don't think excessive academic elitism is one of them. If anything, I think they're quite a bit better about avoiding that elitism than the average you'll find among organizations of comparable wealth and power (compare top law firms, leadership of more traditional companies, politicians, etc). This isn't my strongest opinion - I've never done any proper formal study of the question - but your impressions contradict my own experience living in that world.
I used to be part of the leadership team for a YC company. Here's where the four YC founders I know off the top of my head went to school: SUNY, Oxford, Epitech, Tufts. Only Oxford among those is truly elite, though the other three are good schools. The one who went to Oxford (Harj Taggar) is a bit of an odd one out here, since they were a partner with YC itself first before founding the company that I know them from. Granted, three of these were founders many years ago, so perhaps this has changed, but one (the Tufts alum) got funding only about a year ago.
I looked up a few other founders that are second- or third-degree connections and found the same: a handful of truly elite schools, a peak in A-tier-but-not-truly-elite schools, and a handful of no-name schools like my own alma mater. I expect that that's a reasonably representative distribution of where you find extremely smart people, and especially smart+motivated ones.
As for this bit:
>> And often those who did get in went to Stanford, Harvard, etc. and don't even have a product - sometimes they don't even know what they're going to build yet.
YC is pretty explicit that they try to pick founders, not ideas, most of the time. That approach is all over the Startup School videos, for example; they don't exactly make a secret of it.
>> but the numbers I've read online that most VCs want to see are not realistic - or if we were hitting those numbers I wouldn't even need a VC, we would be able to fund our own expansion.
Growth demands, both realistic and unrealistic, are a much fairer criticism IMO. They're the primary reason I haven't sought VC funding personally (from YC or otherwise).
Out of curiosity, I'd love to know more about what you're doing - I checked your HN profile but it didn't have a link or anything.