I'd say instead that a PoC can be negatively affected by racism that isn't the result of a racist act by someone else. This is why it's so easy for people who don't experience it to minimize or otherwise feel that racism is less of a problem than it actually is. We don't feel like racists and we (probably correctly) assume that others are mostly like us. And so if we and most others like us aren't being racist, racism must not be much of a problem.
Aside from the property values example someone else listed, here's another one. If a well-off community decides to impose a tax on their residents to help fund the local schools, that doesn't feel racist. Everyone is just trying to ensure a quality education for their children. But what that does is allow the underfunded public school system to remain underfunded. If everyone's children were receiving substandard educations, there would be more pressure on the state to fix the situation for everyone in the state. But when only the poor communities are underfunded thanks to the wealthier communities kicking in those extra funds, you end up with those poor, primarily PoC students having lower-quality schools.
In that way, overt racism from previous eras created a status quo that, through non-overt racism, gets perpetuated going forward.