I absolutely appreciate the scale of the problem, and the adversarial nature of peer reporting. My day job has those same characteristics.
Millions of games per week * 30 minutes per game * avg lines of chat per minute = manageable w/ a proper streaming architecture
Especially when you have access to a massive, perfectly-scaled, distributed edge compute system. (i.e. running minimal, performance-optimized models on users' opponents' clients to do the initial detection / filter / compression pass)
But my point is this is fundamentally an economic problem, given current state of the art, not a technical one.
Companies are looking for pure-technical solutions because they're cheaper, and then complaining that it's a hard problem because they're unwilling to properly fund hybrid systems until state of the art can deliver.
ML is a first order approximation of human ability, not a magic unicorn that gives you exactly what you want. Thats the definitions of engineering: how do I build a system that fulfills my requirements from the pieces I have, not the pieces I wish I had?
So I don't feel much pity when companies allow toxic user bases to flourish because it's cheaper than funding solutions.
* Above intended in no way to belittle the awesome work folks are doing in the space with ML detection. But sometimes as engineers we need to admit when management is making unethical choices for financial gain