Earlier quoted context omitted.
I don't think I've seen anyone right-of-center have anything really bad to say about Scott. I doubt any of those outlets (Breitbart, etc.) would want to do that to him. This may itself be reason for some people to distrust Scott, except that he's probably done more to bring people to a moderate or left-of-center position on some topics than all the people shouting "racist!" combined.
I think this is just because the right is on the cultural defensive right now. Most people are really bad people; they don't process liberal principles intended to protect the powerless (on any dimension) as anything but a hollow tool, to be used when they're being protected and ignored (to the extent possible) when they want to crush their enemies. That is to say, those who are culturally out of power _need_ to act…
The rationalist/secularist/etc. community was around back then, and that didn't really happen. I agree that a limited incumbent effect does apply, but it seems quite clear that liberal norms and principles really do get more respect on the (non-extreme) right than they do on the left.