The problem is that, apparently, abusing people wholesale is profitable.
Drug dealing is profitable. Chattel slavery was exceptionally profitable. To allow profitability to be our measure of permissibility is to sacrifice civil society at the altar of enterprising tyrants. Economics should never be a substitute for ethics.
[EDIT: Added after that one person upvoted this comment.]
Slavery was big in the south because there wasn't enough low-skill labor to feel threatened. The north didn't care for it as much because a higher percentage of labor was what we would call semi-skilled or skilled today. That labor was FREAKED OUT about the idea of slaves taking over their jobs and they were able to organize before being eaten by the capitalist leviathan.
I'm pretty sure there were more abolitionists in the north than the south, I don't think your average northerner cared about the plight of southern slaves other than the institution being a threat to their livelihood if it moved north.
If you were making the assertion moral concerns or ethical behaviour eventually influenced american capitalism, I disagree. The capitalist monster acts "moral" or "ethical" because at the current time, to do otherwise is invite political dissolution. I fear the shadowy cabal of capitalist masters will move to reinstate chattel slavery. We did not respond with outrage when red-lining disenfranchised large portions of the populous or when usury was slipped back in with high credit card APRs and payday lending. We are asleep.