I don't think Einstein was being that nice, he was essentially saying that intelligence and education were not guarantees against bigotry and bias.
Aristotle proposed in Book 1 of Politics that certain human beings were natural slaves, as different from the free citizens of Athens as human beings were from beasts. He argued that barbarians deserved enslavement because they lacked souls.
Back in 1550, Aristotle was marshalled in an important debate on race: http://en.wikipedia.org/wiki/Valladolid_debate
While much of the debate was around the notion that Spain had the right to conquer the Americas, a major part of Bartolomé de las Casas' debate with Juan Ginés de Sepúlveda centred on the rights of Native Americans. Both men were humanist theologians and well versed in classical works, so Aristotle's idea of Natural Slavery was central to their cases.
De las Casas argued that native Americans were worthy of the same rights as European Christians because they were rational human beings with souls, not natural slaves. Years later, De las Casas expanded this idea to a universal statement that no human being regardless of race deserved slavery.
Sepúlveda took Aristotle on face value and argued that Native Americans were natural slaves and as such didn't deserve to be treated as rational human beings, he made the odious argument that enslaving them was a paternal response which improved their lives.
It's important to note that the debate on the shared humanity of the races and their human rights is an argument that is at least half a millennium old.