I agree that the degree of possible harm is important here. But I also think that if we start measuring that, then we have to compare the actual positions people hold to the one you ascribed to your hypothetical outspoken Nazi-like person. If someone who advocates murdering trans people were placed on such a board I would be a lot more up in arms than in the hypothetical Nazi-like case. For a number of reasons, including the power dynamic, but not limited to that.
But that's not what we're talking about here, either for the particular person on the ex-board or the overall population dynamics: 30% of the population is not in favor of murdering trans people, and neither was anyone on Google's board. To get to the 30% number I think there are two options: 1) reduce the level of disapproval to the point where you in fact have a meaningful fraction of the population (not 30% by any means, but not negligible either, and including some members of congress, which I realize is much more acutely true for the trans case) with an equivalent disapproval level of Jews, or 2) define any expression of disapproval or concern with complications at all as an existential threat. It seems to me that a number of people do the latter in practice, which is why we end up with comparisons with the hypothetical Nazi-like.
Just to expand on this, I really do think there is a vast difference between 1) people who acknowledge that trans people exist and are "legitimate" in whatever sense one cares to think, have concerns about trans women's participation in women's sports, and want to figure out how and whether that can be made to work reasonably, 2) people who just wish trans people as a concept would disappear because it would make everything so much simpler, and 3) people who advocate violence against trans people. I don't think my viewpoint is universally accepted, and there are various instances in this very thread of comments conflating various positions on that spectrum.
Finally, I agree that if this board is not a serious attempt at ethics oversight and is instead just a PR stunt, then there's no point in worrying about a diversity of viewpoints and all that; Google should just appoint whoever will score the most brownie points in whatever status competition they think they're involved in. But I do think that external oversight of AI ethics to prevent echo-chamber effects is quite important, and I'm disappointed in the lack of such, whether that's because there's no board or because there is a useless board.