Well, two things:
1. I think few companies are out there "hiring by race or gender". At most companies I know, affirmative action means hiring by skill, but making sure at least XX% of the hiring pool are women, black, etc. This really just means your resume is more likely to get seen and maybe you're more likely to get a first round, but not that "you're being hired by race or gender".
2. You could argue this is still unfair (although as previously stated, you're still hiring people who earned it based on skill). Unfortunately not doing anything would me more unfair, specifically by stagnating things in the problematic state they are now [1].
In the macro-historical context, white Americans brought African slaves to their country, and actively deprived of wealth and knowledge, right? That was a societal, conscious commitment to push these people down.
Now that they're free but seriously lacking wealth and behind on knowledge too, they're expected just create that wealth and knowledge themselves. Thanks to "unfair" institutions like public education, this is going to happen regardless, but very slowly unless there's a societal, conscious commitment to push these people up.
[1]: Having a huge chunk of your society be alienated not only socially but also economically leads to conflict, could worsen the gap instead of improving it, weakens social and cultural ties, etc.