I agree as someone from a poor family that went to a tiny school in one of the worst states in terms of education. I was lucky to be accepted to one of the top schools in the world for a combination of grades, near perfect ACT scores, and athletics. I worked hard without the benefit of any tutors and not much thanks to my teachers since most of them taught multiple subjects, most were family members of the school's founder, and they offered no advanced or AP courses. I'm also extremely grateful that my entire tuition was covered by financial aid scholarships because I can't imagine the debt I would be in otherwise as my parents certainly could not have afforded the tuition. One year's tuition was more than their combined income.
I was the first in my school to go to a top 10 college in over 20 years, so they made a pretty big deal out of it. I was also one of two people in the entire university from that particular state during my freshman year with the other student having gone to the top private high school in that state (~30k/yr for tuition).
In other words, hard work and performance on the ACT was a huge contribution to being accepted to the schools I did, although I do contribute athletics as an equal contribution since it gave me the benefit of early admission. Then again, if the SAT was a requirement, I probably wouldn't have been accepted because I didn't do nearly as well on that one.
Personally, I think it should be a combination of academic performance and other more "holistic" measures. The particular university I went to really emphasized being a well-rounded person and are known for rejecting students with perfect ACT/SAT scores because their application didn't reflect any particular drive, motivation, passion, whatever.
Just because you can test well and got good grades does not necessarily mean that you'll thrive in such a competitive environment and make the contributions that the school loves to flaunt. They also want to maintain their graduation rate and don't want people that dropout or transfer from getting overwhelmed (or being entitled because that attitude got stomped out quick). Despite being someone that has always thrived in such a high pressure environments, I admit that things got overwhelming for me pretty often and affected my mental health to a great degree. The stress and imposter syndrome took a major toll on my mental health, academics, and athletics. It got so bad for some that they didn't even survive (suicide).
I didn't do nearly as well in university as I had hoped in any facet, but I don't regret it because it made me stronger and ready for the world. I'm happy with the success I've found after years of working hard even after graduating. However, I do often wonder if I would have been more successful at a school that was less high stakes.
----
For those potentially applying to college and as cliche of advice as it is, I advise really emphasizing personal hardship and how you've overcome it in your application essays. Basically, demonstrate that you recognize your weaknesses and overcome them and also demonstrate that you don't turn away from obstacles.
Give the admission board the perception that you are constantly progressing, not going backwards or maintaining your current progress, and that nothing is going to stop you from doing so. Excellent academic performance is often good supporting evidence of that, but good grades and test scores by itself doesn't necessarily prove that determination. You should provide as much evidence of that as you can, whether it's extracurriculars, succeeding despite social and economic disadvantages, major life events, natural disasters, etc.