As someone who loves to hire people with different backgrounds and experiences, don't see you being included in diversity hiring discussions as a negative. Don't see it as "Oh, they just want the black guy's opinion". Yes! They very much do! They also want to know how it all is perceived by you. You should tell them this. We, as humans, should celebrate our diversity. We should make it a culture where it's seemless (as you put it, "able to be seen"). We should NOT make it hush hush. This is how things become swept under the rug and issues not addressed. The fact that you are here, discussing this, instead of with your manager and (hopefully) your director is a sign that they are missing the point. Diversity is great. Inclusion is better. Participation is best. Go bring these issues up with your manager with ideas on possible solutions. You can't complain about not having a better system without presenting a better system.
That said, it obviously feels to me like you have some sensitivity issues because of this, probably justifiably so. One of the cool things about tech is that when it comes down to it, it's the output that makes the "man" and not the skin color or their name or their religion or nationality. If you can write some clean, tested, code, I'm all for it! The fact that you are YOU, is even better! Why? Because I don't want a room full of white guys that just agree to what I propose. I want females, I want PoC, I want refugees, I want "I pulled myself out of poverty", I want "I would use this", I want "I care about the outcomes". Not in a cult like way, but in an empathetic to the customers problems way.
Some companies have put metrics around this. Hire so much diversity... hit 40% something-or-other... Ignore these false metrics that are doomed to failure. They aren't approaching the problem right. Let them do what they think is right unless you have a better solution. Keep just doing you and look at their efforts as a good, not a negative, even if their methods are naive.