I think that this is true. This is also true that you can't understand what it is of being powerful or rich when you're not. In a general way, it is very difficult to imagine living in other people's shoes, we live on the same planet, breathing the same air, we have 99.9% the same DNA, but our experience of life and our environment can be widely different.
I agree what you say, although I might add that the harm potential by a "not-understanding rich of the poor" is much greater than a "not-understanding poor of the rich"
This is in no way blaming those in poverty for not magically knowing how to ace job interviews or detect which risks are worth it, but pointing out that in both cases, X not understanding Y hurts the poor more than the rich.