You're literally describing what microaggressions are: Small interactions, any one of which is harmless and without ill intent, which accumulate over the course of the day, week, or lifetime as an atmosphere of pervasive discrimination which is completely invisible to those responsible for creating it.
Microaggressions are insidious precisely because they're so innocuous; they let well-intentioned people participate unwittingly in systemic oppression that they would never consciously support.
If you're noticing yourself get a bit defensive about this idea, forget the word itself and just frame it like this:
You're a well-meaning person, and you want to avoid causing unnecessary harm with your words. So when you're around people who have really different life experiences than you, how do you make sure you don't cause harm without meaning to?