Then we have social justice for fat people, dumb people, lazy people, people with anger problems, people with personality disorders, criminals, and so on. Where do we draw the line?
I argue we don't draw the line at all. Rather, we treat everyone with compassion and respect, and find / judge everyone on their true strengths and weaknesses (vs. those characteristic of whatever group they belong to).
Discrimination is OK when the discriminating trait actually matters - i.e. you wouldn't let an idiot or a hot-headed person control a nuclear warhead or perform brain surgery. Similarly, I have nothing against diversity, but I think portraying people of many ethnicities in a movie whose setting would not have people of all ethnicities is weird and something our current culture gets wrong. But you should not be rude to an idiot or angry person, just like you shouldn't avoid putting ethnic people in general movies (and make sure you actually know your setting, e.g. I recall controversy where black people were in a movie on ancient Greece, but black people actually did exist in Greece at the time).
(Also I still think people who commit crimes and are dangerous should be locked up. The point is they are locked up to protect and deter others and possibly reform them, not just for intrinsic punishment.)
The issue with the current attitude on social justice is that most people don't fall for prejudice or stereotypes, but don't understand why. You don't just avoid stereotypes because they're often wrong (though a lot truly don't line up with statistics), you avoid them because you don't judge people on their surface, people aren't always what they look like, and most people fail at least one stereotype of a group they belong to if not many more. You don't treat someone with respect because they're black or female, you treat them with respect because they're a decent human. Similarly, you don't mistreat someone whose short or fat or even has something wrong with their head which makes them occasionally annoying. But lots of "social justice advocates" don't realize this and hate towards sides which aren't explicitly "protected".
EDIT: There's also the practical situation where you can't save everyone. In that case unfortunately I think whatever disadvantaged or stereotyped groups aren't protected by the current society are out of luck. But while you can't always give everyone tangible aid or a thorough evaluation, you can at least treat everyone with respect and common decency.