How Reliable Are University Rankings?

Not true of the UK, France, Korea or Japan. I don’t know if they care about what are basically league tables but any educated citizen would find it no more difficult to reel off five to ten of their best universities than an American would to list Harvard, Yale, Princeton, Stanford, MIT etc.

I can see how it would look weird from a German, or possibly a Dutch or Nordic perspective but a deliberate effort to bring up the bottom and limited attempts to raise the top is extremely different from the US system, where there are large and growing returns to excellence.

Also the US elite just have smaller enrollments as a share of population. Oxbridge, the grandes écoles or Korea’s top three enrol 1-2% of yearly students. The top ten US universities maybe a tenth of that.

It's very similar to sports teams.

The tops teams don't vary all that much from year to year. They might change positions a bit, but it's rare for a new team to show up who hasn't shown up before.

They essentially have to get lucky and get someone who is super talented into a long term contract, who then attracts additional talent.

Or they have to invent a whole new way of doing things, like how the Oakland A's invented a whole new way of recruiting using math (which incidentally no longer helps them because all the top teams adopted it and are back on top again).

It's all about network effects.

Care to elaborate? Did you have any problems after graduating from the "top" university?

Sometime not too long ago there was a blog post or article covered on HN, about data showing student scores on some academic achievement measure over time. If I remember, all students generally improved on this measure throughout undergrad, and there was some average tendency for universities with higher average entering standardized test scores to have higher scores, but the effect seemed to be mostly driven by the students. That is, it wasn't that some schools accelerated scores more, it was that they just tended to select for higher-performing students. Also, the signal provided by school, if I remember, was relatively weak. The author was basically arguing that universities in general provide some educational value (or students just learn over time; there was no control of students who didn't go to college) but that the ranking was by and large just a signaling thing.

I can't find it now though because all the search terms I can think of just return a bunch of other stuff related to universities and students.

I'm mostly on board with the critique of university rankings, but I really don't buy this specific critique. It's a strawman.

Here's what they did: they took the top research universities and the top liberal arts colleges. They then observed that the rankings can't distinguish between the top institutions within each class, and that you can also flip which of those two classes is preferred.

But, in what sense is it even meaningful to compare Amherst College and Harvard [1]? Those are just enormously different types of institutions. It'd be like creating a ranking of "best cars" that includes the top 5 sedans and the top 5 trucks, and then observing that you can jiggle the rankings to get anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question. US News and World Report, for all its problems, does at least get this much right! they break down institutions by "type" and then rank within type.

Additionally, just because rankings are noisy and easy to game locally doesn't mean they are inaccurate or easy to game globally. Two institutions within 10-20 slots of one another are probably pretty similar and rankings aren't particularly helpful / are easy to game. But the #100 liberal arts college is probably not as good as Amherst, and the #78 National University is probably not as good as Harvard, and no amount of gaming is going to change that.

Rankings are indeed noisy and inaccurate and easy to game. But this particular article is not a compelling demonstration of that fact.

[1] For non-US readers: Amherst college (not to be confused with UMass Amherst) is in a class of peculiar institutions that are fairly unique to the USA as described here: https://en.wikipedia.org/wiki/Liberal_arts_college TL;DR: they're basically the diametric opposite of super-charged research universities like Harvard.