Yes, the vast majority of people, including me, would disagree with that.
However, why is somebody holding these opinions a show-stopper?
One could argue that that's because in many cases somebody holding these opinions is a "dog whistle" for the person actually being a paedophile. In the case of RMS, though, we have no evidence that he actually is a paedophile, despite him being pretty thoroughly scrutinised over the years. We also know that he has a need to question everything and write about where he disagrees with society, which he presents in excruciating detail on his website, so it's not that he has some sort of fixation on this topic.
I'd argue that in an ideal society we should encourage thoughtfully heterodox people, since once in a while they'll invent something revolutionary like free software and copyleft, while them also sometimes holding some very weird opinions is not much of a cost.
In the non-ideal real world, once RMS came up with his revolutionary ideas, he probably should have stopped talking about his most fringe ideas, so his resignation as president of the FSF, for instance, had been sensible and over-due.