Will these systems be prone to false data? I.e. if an organization fighting for free speech would create software generating false personas who create false messages, and other events, would they know any better? For example, it wouldn't be too hard to generate hundreds if not thousands of fake Facebook, Instragram, Airbnb, Coinbase, Uber, Gmail, Amazon, etc. accounts doing "stupid things". Like ordering stuff and ca…
That wouldn't matter. These companies would still have real data for actual people. And it's not easy to create tons of fake accounts for services like Coinbase because they have strict identity verification. Any account that doesn't complete it obviously isn't really a real account.
Without very careful system design, that itself could be used to corrupt data.
Suppose a thousand obviously fake accounts are made and use the phrase "giraffe-eater" (to coin something random-looking) in messages.
If the system is designed as a fancy nonlinear regressor, then it doesn't have to obey causality -- observing P("giraffe-eater"|fake) = 100% will increase the modeled prediction of P(fake|"giraffe-eater").
Worse yet, if the system is in fact being built in a non-interpretable way (throwing everything into a random forest or deep neural network), then it will be impossible to prove anything about how the system makes its judgement. Ultimately, these models are at risk from anti-discrimination statutes if (for example) they can key on use of dialect or other protected-ground-correlates.