The Economist's US college rankings

If one school tends to attract students interested in making a lot of money, and another school tends to attract students interested more in academics and graduate school, can one really conclude that the former adds more value? Perhaps, but certainly not without taking into account the different student interests and inclinations towards money-making.

They do control for field of study, but I don't think this is sufficient. A school that attracts a lot of STEM majors who go to graduate school instead of industry is pretty much going to get destroyed in this study.

Yes, I did. In suggesting that their methodology is "debatable", they are indeed hedging their bets. But they are still advancing the theory that, after looking at the data, one could plausibly accept their model.

My "anecdotes" consist of four years as an undergraduate at a university (Rice) rated in the bottom 0.5%. I know damn well that Rice prepared the vast majority of students to be extraordinarily successful in the career paths they chose. I also know that my education was tremendously valuable for its own sake, and that the same was true for my classmates.

And as an academic now teaching at the University of South Carolina, I can only wish that we offered our students what was offered at Rice.

It is true, as a Rice graduate I am biased. But nevertheless I do not believe that I am stretching when assert with confidence that, according to any metric which measures anything worth measuring, Rice lies somewhere in the top 99.5% of universities in the US.

For example: Why should CalTech get hurt for being near L.A.? They're basically arguing that you're better off going to a school in the middle of nowhere because "hey, for being in such a crappy location, you did pretty well!". In an absolute sense you are better off going to CalTech, it's just that they might not leverage their advantage as well as some other schools.

Not that they even show the last point -- it seems unlikely that the true model is linear (I'm guessing they used linear regression). For example, if the true model is closer to a sigmoid, then schools at the high end suddenly get unfairly penalized and schools near the low end get unfairly boosted.

Finally, the statistical indicators are equally misleading. I can obtain an R^2 of 1.0 just by including indicators I[is COLLEGE_NAME] for each college. While that might not give you significance, the point is that getting good prediction is meaningless.

I think what they really want is to restrict to predictors about the students. So, given that you're a straight A student with a 2400 SAT score, what would you expect to make coming out of each school? This at least tells me something about the added value to me of going to a certain school. (This approach is still prone to bias, but in the opposite direction -- there's a chance that the straight A student with a 2400 SAT score going to community college may have been smart but unmotivated, which might correlate with lower salary.)

Edit: Here's another concern. They're claiming to have a model for "expected" earnings:

earnings = A * (college covariates) + b + error

but they can't distinguish between model error (i.e., error because their model is misspecified) vs. the school variation that they are trying to capture.