I'm confused. The authors cite several instances of research done at other institutions as the main evidence that SAT/ACT scores are predictive of overall success. At that point, the a priori assumption is that Dartmouth would behave the same way, which they then go on to show. I agree that they could have fully analyzed graduation rates and full-term GPA, but other studies have already done that. I do believe their analysis is thorough enough (looking at first year GPA) to adequately show to me that there is absolutely no reason to believe that Dartmouth would be any different than the other institutions at which much larger / more rigorous studies have been done.
I mean... do you have a full double-blind study done when you're sick to determine the right dosage for you, or does your doctor consult an existing dosing study, make sure nothing is substantially different between you and the study group, and then decide upon an appropriate dose. There's a certain point at which over-analyzing makes little sense. The authors here ARE NOT engaged in research. This is NOT peer-reviewed. It's a letter elucidating an opinion by researchers in the field based on their knowledge of previous literature in the field and then a surface-level analysis to make sure that those results ought to apply to Dartmouth as well.
Moreover, you're hyperbolizing yourself. Namely, you criticize the title, which is not the author's title at all. Their letter is titled "Report From Working Group on the Role of Standardized Test Scores in Undergraduate
Admissions"
The authors main claim (on the first page after niceties) is that ".Our overall conclusion is that the use of SAT and ACT scores is an essential method by which Admissions can identify applicants who will succeed at Dartmouth." This is not "SAT / ACT scores are highly predictive...". It's an entirely different statement. Let's not confuse HN editorializing with the contents of the letter.
> The title would have you think that if you know a student's SAT scores, you would be good at predicting their outcomes at Dartmouth. The data shows this is wrong: if you only used SAT scores, you would almost always be wrong in predicting how well a student would do.
Except if you read the text, that's not at all what the authors claim. The authors do not claim the title (again... that's HN's title, not the paper's). The authors say:
> In short, consideration of SAT scores allows Admissions to identify applicants who will thrive academically at Dartmouth better than does the use of high school GPA alone.
which is true. Even if you don't think it's a useful predictor, it is a more useful predictor than GPA. This is a reasonable contrast because in the absence of test scores, this would be the only data available.
As for the other critiques. I agree that Figure 1 is not useful and shouldn't have been included. However, the table at the end is pretty straightforwards and provides essentially the same conclusion as above, which is that SAT is better than GPA. Moreover, just because the figure they included is bad does not mean the data they studied was. We'd need to know more, not wholesale reject their conclusion.
The authors then go on to argue that other papers agree with these conclusions (they do), and that alternates to both SAT and GPA (letters of rec, etc) have been shown to exacerbate inequality while having very little power to predict how well a student will do.
> The fact that there is a linear relationship means nothing statistically.
It does... in fact... mean that there is a linear relationship in the observed data with the given levels of significance. I'm not sure what exactly you're going at here. I agree a scatter plot of all the data might be more useful, but honestly, given the plethora of other work on the subject published by so many disparate institutions of higher ed, I really have no reason to believe dartmouth would be much different. So while the included figure is not great, I'm guessing that if we saw all the data the conclusion would probably still be the same.