Society is only the collection of individuals, you are ascribing features to it, and implying some sort of moral order when there is no such thing.
The society is not judged by anybody and there is no universal moral rule that distributes outcomes to demographics. There are only people interacting with other people.
As a human, I am concerned about the personal freedom of individuals. I will do whatever it is in my power to make sure any human being truly has the individual liberty of pursuing life and happiness. I know from personal experience how difficult is to fight for one's freedom alone, so ethics compels me to join the fight in anyone's struggle. I instil in my children moral guidelines that I believe preserve individual choice and individual freedom because that is the world I want to live in and the world I consider just for my children.
At absolutely no point groups of individuals have intrinsic moral rights, and the whole idea of gradually shifting vocabulary from individuals to groups until established terms such as "gay's rights" and "women's rights" form is nothing more than a subtle effort of obscuring the original drive behind class struggle. Suddenly established language shorthands become legal concepts and the whole ethical struggle about ensuring individual freedom becomes a debacle about demographics.
It is bad to discriminate black people because it restricts the individual freedom of every black person, not because there is some intrinsic moral charter that codifies blacks. If I were a black person I would not want to be discriminated against, so my ethics make me fight against any specific examples of discriminating against blacks.
It is bad to discriminate gay people because it restricts the individual freedom of every gay person, not because there is some intrinsic moral charter that codifies gays. If I were a gay person I would not want to be discriminated against, so my ethics make me fight against any specific examples of discriminating against gays.
It is bad to discriminate women because it restricts the individual freedom of every individual person of feminine sex, not because there is some intrinsic moral charter that codifies women. If I were a women I would not want to be discriminated against, so my ethics make me fight against any specific examples of discriminating against women.
And so on ad ad infinitum.
However we can talk about black people, gay people, or women purely as groups of individuals where the rights of the group (if we can talk about such things) is the collection of the individual rights of its constituents. It's an extrinsic concept that arises from the way we look at groups through generalisation. We haven't imbued any distinct, intrinsic right to the group. The group does not exist outside its members, and its members are concrete living human beings, not abstractions.
Forgetting this crucial fact is a insidious strategy to make people take decisions that infringe upon individual freedom in the name of the freedom of some group.