I don't know if this is a downside.
First a bit of background. I come from India where they practice reservation which I think would be considered much "worse" than affirmative action over here. The way the system works is that a certain percentage of seats are reserved for people from certain (usually lower castes). So for my engineering examination, I wrote a common exam conducted by the government along with about 80,000 other students from my state. Among the 80,000 about 35k or so did well enough to be given a "rank". The rank means exactly what you think it means.
Now, students are allowed to choose the college they want to go to in the order of their ranks. However, reservation means that people like me, who don't "have a reservation" have to choose seats from the general quota. The general quota was, believe it or not, 20% of all seats. Out of my class of 140 EE students, we had 29 students who came through the general quota. About 15 more seats were available to students who were willing to do without the government subsidy.
Side note: Because my education was subsidized by the state government, I paid about $300 dollars a year in fees. The ones who went without the subsidy paid about $1200 dollars a year. All
All the remaining 100 or so seats were reserved for people from one or another of the reserved castes. For people in the general quota, the lowest (highest numerically) rank admitted to my department was 114. For a person coming from a reserved quota, the lowest rank admitted was 29xx.
At the time I felt this was grossly unfair. I busted my ass to be ranked in the top 100 among 80,000 candidates to get into this place and these guys were admitting students who weren't even in the top 2000 simply because they were born into the right family.
Over time, I've come to realize that I enjoyed a great many privileges these other kids did not. I had sporadic access to the internet, I had parents who were reasonably supportive of my studies, I knew English really well. I didn't get the 90th rank simply because I was awesome. I got it because I was lucky enough to be born with (some) privilege in a third-world country.
Many of my classmates from disadvantaged backgrounds have gone on to become competent engineers. I realize now that they fully deserved their chance.