Earlier quoted context omitted.
but in your direct quote, it clearly says "not-rich, not-white", so clearly the direct quote was not talking about "rich blacks"
The phrase can be interpreted two ways. To simplify this comment, when I say "black" I mean "not white". Easier than to keep typing non-white, etc. You have a quadrant of four categories: 1. Rich and white 2. Poor and white 3. Rich and black 4. Poor and black You are interpreting the statement to refer to "people who are not (rich or white)" - which with Boolean logic expands to "people who are not rich and not white…
This is not just a matter of two equally valid alternative explanations: a reasonable reading of the original sentence "not poor, not white" (along a minimal amount of thought about the context and subject matter, e.g. prisons clearly are not full of rich blacks) would have easily led to the other person realizing that it was category 4 that the parent was talking about, not 1-3.