On the contrary, test scoring is the most objective thing we had to evaluate students based on the diligent work of a company whose entire business is maintaining that perception of objectivity. Sure, it's "the perception of", but if you think there aren't smart people analyzing college board exam scores across the country for signs of skewed testing so they can write an expose on it, you're far from reality.
In fact - the line that you're walking is the reason there was briefly a suggestion of an "adversity score": https://www.google.com/amp/s/www.businessinsider.com/college...
Which is the first time that's been considered, because in the current climate - working extremely diligently to try to ensure that no particular student or group of students has an advantage, some students do have an advantage. They are smarter, or have better parenting, or better education, or their parents simply have more money to throw at tutoring
You can say "that's unfair! This is about selecting the best students and locks smart people from worse off backgrounds out of university!"
It does the second thing, sometimes, as a side effect - but it is also true that those students are still better students, regardless the path dependence.
Meanwhile, universities are enacting racial and gender preference schemes,
Cites: https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...
https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...
Additionally, the primary winner of the policies this line of argument promotes are white women - with the primary loser being Asian men. Asian students in particular are bringing the suit against Harvard for these practices.
So while your epistemic humility ("is this REALLY objective? How do we measure that?") is well intentioned, it's the exact sort of reasoning used to cover up a strict ideology of "more black good, more women good, less white good, less men good" in universities, and the actual behavior of universities shoe that.