AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
blog.wikimedia.org
AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
1–10 of 52 posts
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#2Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#3Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is.
To complete the circle, it would be nice to know how it performs according to this:
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#4Instead author gives us some insight into his/hers source of inspiration:
> A feminist inspiration
> “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost)
And few lines below we have this gem:
> While artificial intelligence may prove essential for solving problems at Wikipedia’s scale, algorithms that replicate subjective judgements can also dehumanize and subjugate people and obfuscate inherent biases.
Essence of not being sexist or racist is not to evaluate people basing on sex or race! Author instead accents that we should totally pay attention to subjective factors (such as sex or race) instead of objective factors (such as quality of edit).
And let me guess whos edits will be targeted with extra scrutiny.. Edits of people who don't buy feminist agenda!
(Disclosure, I'm egalitarian and MRA; While I fully support women rights, I just believe that modern feminism does not have anything to do with equal rights for anyone. This post nicely fall into this pattern.)
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#5Is there anything about the false positive rate of this system? Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is. To complete the circle, it would be nice to know how it performs according to this: https://en.wikipedia.org/wiki/Precision_and_recall
Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a work in progress and hope that it will improve over time. But hey, it is the first time a vandalism detection tool is deployed in Indonesian Wikipedia. :)
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#6This is very peculiar. New tool is introduced, which joins a set of existing tools (bots) which address the same problem. Some comparison of quality and robustness is expected. Instead author gives us some insight into his/hers source of inspiration: > A feminist inspiration > “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost) And few lines below we…
Suppose, for example, that a user repeatedly rejected edits for anyone with a noticeably feminine username. Wikimedia wants to be sure that their algorithms don't take this person's rejects to be valid data. Again, they don't really say how they'll do this, which makes the whole section rather confusing.
It's a valid concern, just presented awkwardly.
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#7This is very peculiar. New tool is introduced, which joins a set of existing tools (bots) which address the same problem. Some comparison of quality and robustness is expected. Instead author gives us some insight into his/hers source of inspiration: > A feminist inspiration > “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost) And few lines below we…
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#8This is very peculiar. New tool is introduced, which joins a set of existing tools (bots) which address the same problem. Some comparison of quality and robustness is expected. Instead author gives us some insight into his/hers source of inspiration: > A feminist inspiration > “Please exercise extreme caution to avoid encoding racism or other biases into an AI scheme.” > Wnt (from The Signpost) And few lines below we…
I agree the "feminist inspiration" part is a confusing non-sequitur. I think you're missing what they're trying to say, though. I interpreted that section as "we're making our algorithms and data open so that everyone can be sure they aren't implicitly racist or sexist". Suppose, for example, that a user repeatedly rejected edits for anyone with a noticeably feminine username. Wikimedia wants to be sure that their al…
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#9Is there anything about the false positive rate of this system? Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is. To complete the circle, it would be nice to know how it performs according to this: https://en.wikipedia.org/wiki/Precision_and_recall
Not really, but one can report a false positives and false negatives here: https://meta.wikimedia.org/wiki/Research_talk:Revision_scori... Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a wor…
That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.
Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits
#10Wikipedia has several perspectives just within the colloquial / perceived mission. Is it for academic integrity? For common education? For the rich and powerful? For the everyman? To record as much as possible? To cull the best knowledge from the stream?
Is it for progressive ideals, which may be opposed by the common majority? Or does it reflect a conservative viewpoint, adopting only what has proven to coalesce in society?
For who's society? What about two sides of a war? Or two equally opposing economic interests (entrenched oil vs alternative energy)?
"good" or "bad" isn't a number...