Earlier quoted context omitted.
Curtis Yarvin is a professed fascist who has written about his belief that some races are more suited to slavery than others. That _does_ matter; it makes large swaths of the potential talk audience (and other attendees) feel unwelcome. How can you engage in polite, respectful discussion with someone who believes that you're more suited to be a slave than a free man?
It's a technical conference, submission accepted on said merits. If personal politics and beliefs are not apart of the talk, I don't care. Polarizing of personal beliefs, unless specific personal actions are documented, is comparable to a witch hunt. At least for me. My own beliefs waver on the spectrum of Silicon Valley, yet I have concerns for outside options I have. But, no, at a technical conference, if personal…
You can't separate the two. Spouses invited to a speakers dinner? What if my spouse is also a man? What if my spouse is a different race? Should we give priority to people of color and women in the speaker selection process? Are trans people allowed in whichever bathrooms they want? Giving a talk on preventing fraud and abuse, where part of your belief system is "women should be free to give opinions online without bullying, trolling and harassment?" How are you supposed to give that talk without communicating a value system?