The tests have been gamed by a test prep industry that extracts money from wealthy families. Colleges aren't admitting students based only on diversity. They are taking kids with excellent grades, volunteer work, etc. and giving them a closer look. It relies on the judgement of all of the student's high school teachers, rather than a centralized corporate assessment.
Poor kids don't get second or third tries at tests that cost money to take. Even the most brilliant students can have a bad day.
Colleges are held financial accountable to enroll students they can retain, so they can collect a full-term tuition. They aren't going to risk giving a valuable spot to a student that won't be able to perform well academically, just because they are 'diverse'.
Professors would rather teach exceptional, motivated students from all walks of life than the wealthy, entitled children that can afford to spend $5000 on a test prep coach.