It's pretty difficult. Racism, like other ideas that at some point were not considered unacceptable in public, is nowadays covered in what we could call Hermetic writing: In other words, instead of saying what they really want to say, a racist says things that imply they are OK with racism.
A way that you can find all over his writings is talk about how people and businesses should be free to refuse dealing with people for any reason whatsoever. It's not quite saying 'I do not like brown people', but instead 'In my ideal society, we are free to discriminate against brown people, or people that don't share my religion'. It's the same kind of rhetoric you'll find in traditional racist groups. That's enough for many people to call that rhetoric racist, but I see how you might not agree.
What I find most amusing is that in that libertarian utopia where people can discriminate at will, you can discriminate people because of their political views, or because you think it'd make some people feel less welcome, and that's exactly what happened here. Having people that defend the right to discriminate at will complain due to discrimination is interesting to say the least.
Either way, I don't think this is an economic decision though. Last year, StrangeLoop ran out of tickets in a few hours. This year, I know they had more companies wanting to sponsor the conference at than they had slots! So Alex could have reacted either way to this controversy, and he'd have done fine economically. This just seems like very predictable behavior given their pro-diversity direction, taken after a few years ago, they had so few women that they had turned all the female restrooms in the opera house into male restrooms, leaving just a single 'family room' in the entire venue for women.
If anything surprises me, is that they didn't vet their speakers before accepting submissions. Rejecting conference talks because of who the person is happens all the time. What turned this into a contraversy is that they rescinded the invitation after making the list public. I would be surprised if, for next year, they don't add an extra step to their process, to try to catch something like this in advance.