There's nothing intrinsically racist about having lot size preferences, but the lot sizes we have in most 20th century vintage zoning codes are in fact racist by design. Minimum lot sizes were a way of keeping Black families from crossing borders from redlined ghettoes into white neighborhoods or suburbs, by preventing people from subdividing lots into smaller, more affordable houses.
Meanwhile: it's perfectly understandable that people don't want to see change in their neighborhood, or that they buy a property in the expectation that everything good about it will remain. But that's not a reasonable constraint for the law to operate under. You do not in fact have a strict right to control things that happen outside the borders of your own lot.
Some community restrictions are reasonable. We broadly agree that it's not OK for someone to open a tannery in the middle of a suburban residential block. Others are not; for instance, neighbors several blocks over will argue that they have a right not to endure extra traffic when our local hospital, the largest employer and best hospital in the region, plans a small addition.
The most important phenomenon here is hyperlocalism. The immediate neighbors of new proposed residential developments will reliably oppose it. They'll also make up the overwhelming majority of those who show up for public comment, because normal people don't turn out to support new apartment buildings built across town. But if you accept that resistance as a given right, you're essentially saying nothing will ever get built.
The muni I'm in has managed to go from 70,000 residents to 50,000 by consistently applying this strategy, so it's not even accurate to say it's about "change", so much as it is about strangling out as many residents as possible to achieve a targeted demography.