Looking across multiple cultures and economies that had slavery, slavery has a negative impact on both slaves and slaveholders.
Of course, nobody cared about the impact on slaveholders while there were actual humans being enslaved, but as we move to a robotic society? This is going to be a huge deal. Slaveholders and multi-generation slaveholding families have a fundamentally different way of looking at themselves and their culture than people who do not own slaves. Once we enter an era where every person is effectively coddled by multiple robotic "slaves" that do their every whim, we're going to be hacking into the human social ecosystem in ways never anticipated before.