Mistral's Shieldstral: 3B open-weights model for multimodal moderation

Most moderated spaces these days rely on moderating based on "civility" because it can be excused with jargon like "creating a marketplace of ideas" which ignores the reality that the scope of discussions selects for who participates in them as much as the way they are phrased - a zebra will be less inclined to participate in a "marketplace of ideas" where a recurring topic of discussion is how zebra meat is best prepared for consumption even though that space might be very attractive to lions and tigers.

But I'm not sure if this is truly accidental. Ever since the advent of online advertising, online spaces have been overtaken by corporate interests. Heck, it's even endemic to "social media" given that those platforms themselves have turned into major corporations or at least were acquired by them. I'm not implying any nefarious intent but "civility" is certainly the dominating factor when it comes to what corporations care about when it comes to content moderation - anything beyond that is largely about what target demographic they're trying to attract and what virtue/vice signalling is optimal based on the current social and political environment (cf. various major corporations demonstratively dropping "DEI" initiatives following Trump's election).

I would also argue that in terms of content (rather than tone), corporations are necessarily also much less tolerant of "far left" issues than "far right": all social justice movements at the end steer towards anti-capitalism because they run counter to the perpetuation of social (or economical) hierarchies. This is why we saw so many tech companies (including those formerly described as "very liberal") shut down their DEI initiatives even before Trump got elected - because the political window had shifted to the point where this had become defensible while at the same time many DEI ideas had become so widespread culturally that these initiatives now became a direct threat to the "(old) white men" running those companies. This had been inevitable but DEI was seen as a necessary marketing effort (both internal and external) at the time, not something truly adopted on ideological grounds. This is also why speakers/trainers promoting "white guilt" were more popular - making your white employees feel bad is less threatening than making your marginalized employees think critically about the structures that lead to their marginalization; you want to individualize the problem, not direct attention to the systems underpinning it.

Another factor is that on social media content is mostly moderated "softly" by the algorithm. This is the content moderation you don't get to see because it can exercise editorial control where the simple word filters can't. The word filters create plausible deniability: they "try" to filter unpalatable subjects but those darn kids are just so clever and circumvent it. Meanwhile the algorithms can be fine-tuned so the topics you really don't want to see discussed stay off most people's "for you" pages - or even so those who would be attracted to them still see them and feel elevated and heard despite actually being isolated into their own echo chamber.

I think the simple explanation is the likely one (the reason I deliberately chose this specific benchmark): the model isn't intelligent enough to figure out use/mention distinctions. It understands Voltaire is discussing injustice, violence, tolerance; but it doesn't understand which side he's on.