One of the issues with the US is not just that certain races are more likely to be poor but they are segregated as well, meaning that there are counties, neighborhoods, and school districts where minorities are concentrated, at levels sometimes approaching 90% or more. And as a result of that, historically the residents of those neighborhoods weren't just economically deprived, but cut off from power, and influence. People lived in dilapidated housing. They couldn't afford to repair because banks wouldn't issue loans. They couldn't afford good schools because schools are funded by county or district in the US and so a poor local population means underfunded schools. They couldn't get good jobs because they had a poor education. All of their friends, family and neighbors were similarly dirt poor. In other words there was a cycle of pervasive generational poverty which didn't exist in the mainstream culture. Even if they scrimped and saved to try to move to a better neighborhood, they were often either entirely excluded on the basis of race, or faced rampant and sometimes dangerous discrimination upon settling in the new neighborhood.
Anyway, so the goal of affirmative action was to try to break that cycle, by essentially awarding the best and brightest from those segregated communities opportunities they could never achieve otherwise.
There was also maybe a sense of society thinking (at least in the case of blacks and maybe indigenous americans) "We owe them because we put them in this situation by kidnapping/enslaving or massacring their ancestors." Whereas with just generic poor people, to say that we owe them anything (as in "we owe them because our economy requires a pool of desparate labor") would be an indictment of capitalism, which is not an acceptable thought pattern in the USA.