Earlier quoted context omitted.
tptacek nailed it with regards to it being historically racial, but I've also seen, to your point, animosity towards anything that isn't a "traditional" or nuclear family. Fear of the different and unknown is a powerful motivator of the people who bring these complaints. Also animosity towards poverty and the belief or perception of material increases in crime and/or traffic. (property owner, many units, yimby, have…
I'm sure in some/many cases there were racial motivations but also SROs that have many unrelated people in individual rooms often become seen as flophouses and are seen as undesirable by other property owners. So they are often prohibited at least in single-family zoned areas.
Single-family zoned areas by definition only have single-family homes, and only some families will be able to afford those homes. The designers of such policies are often well-aware that the only families affluent enough to purchase or rent a home in such an area is of the preferred race (thanks to the strong racial correlation to socioeconomic class, as well as historic explicit racist policies like redlining, employment and education segregation, etc etc).
To be clear, I don't believe that single-family zoning is necessarily racist, especially with the continued democratization of the middle class. Single-family zoning is bad for a lot of reasons. Its just that it is absolutely the case that historically single family zoning was deliberately used in several cases as a racist policy, but in such a way that it would pass muster in court.