There are advantages to diversity by itself. These have been well documented. So setting aside any concern about excluded groups, it was a good business decision to aim for a diverse workforce.
But taking those concerns up again, almost no one perceives their own biases as biases. It's like you don't perceive your own accent, people of different racial backgrounds look similar, you can't smell your own breath, etc. So being a biased person feels just like being an unbiased person, but it makes you make biased decisions. A person against whom you harbor a prejudice applies for a job. Your bias causes you to discount their better qualities and double count their worse qualities. So you hire someone else. This hurts both you and the person you didn't hire.
A policy that works against well-known biases, if implemented correctly, achieves three ends: it reduces the harm to the person biased against, it reduces the harm to the company whose hiring policies are distorted by bias, and it helps the company by producing a more diverse workforce, which is an end in itself regardless of whether you care about harms to others. If implemented correctly, diversity hiring targets can produce the effect of hiring without prejudice despite the prejudice of those hiring.
If you are crossing a river and the wind blows you off course, you don't head to your goal but to the side. The net result is that you reach your goal. The diversity targets are just tacking against the wind.