Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

blog.wikimedia.org

Post reply on HN
Print view View original

11–20 of 52 posts

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#11

Post by tptacek »

cowardlydragon wrote: post #10

As referenced in the article, what is "good" and "bad" is dependent on perspective. Wikipedia has several perspectives just within the colloquial / perceived mission. Is it for academic integrity? For common education? For the rich and powerful? For the everyman? To record as much as possible? To cull the best knowledge from the stream? Is it for progressive ideals, which may be opposed by the common majority? Or doe…

"Good" simply means "unlikely to be reverted". The objective is to save time for editors.

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#13

Post by logicallee »

tptacek wrote: post #12

I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.

That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#14

Post by tptacek »

logicallee wrote: post #13
tptacek wrote: post #12

I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.

That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?

That's all the model needs to do to be valuable.

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#15

Post by tptacek »

jacquesm wrote: post #9
kenrick95 wrote: post #5

Earlier quoted context omitted.

Not really, but one can report a false positives and false negatives here: https://meta.wikimedia.org/wiki/Research_talk:Revision_scori... Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a wor…

> Not really That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.

It's not actually doing reversions, right? It's just scoring edits. Wikipedia editors keep hot-lists of articles they watch edits for, or patrol lists of edits or page creations; ostensibly, all this needs to do is sort those lists of edits.

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#16

Post by kenrick95 »

tptacek wrote: post #15
jacquesm wrote: post #9

Earlier quoted context omitted.

> Not really That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.

It's not actually doing reversions, right? It's just scoring edits. Wikipedia editors keep hot-lists of articles they watch edits for, or patrol lists of edits or page creations; ostensibly, all this needs to do is sort those lists of edits.

Yup, it's just scoring the revisions. Currently there is no straightforward way to feedback to the tool for learning false positives yet.

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#17

Post by kenrick95 »

tptacek wrote: post #12

I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.

There are already bots and other tools in operation on English Wikipedia for fighting vandalism automatically and semi-automatically. They're mentioned in the blog post (Cluebot NG, Huggle, STiki).

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#18

Post by kenrick95 »

logicallee wrote: post #13
tptacek wrote: post #12

I'm surprised it took this long. Wikipedia has what would seem to be an excellent and straightforwardly encoded set of training data: a constant stream of edits and reversions.

That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?

They also have a model to predict a quality of edit, based on the WP1.0 quality model that has been used as a guideline for some time.

https://meta.wikimedia.org/wiki/ORES/wp10

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#19

Post by logicallee »

tptacek wrote: post #14
logicallee wrote: post #13

Earlier quoted context omitted.

That's not training a model to predict the quality of edits - that's training a model to predict the reactions of admins/editors :) Perhaps a better title is "AI service models Wikipedia editors"?

That's all the model needs to do to be valuable.

Well, suppose something told me what I would decide if I gave any matter 15 seconds of consideration. It would be an invaluable tool I might use often, for example for sorting and classifying, throwing out trash and so forth. For example, it would be a fantastic spam classifier, since even after 15 seconds there is no doubt. I could use it to get through hundreds of spam messages easily, or delegate things to appropriate departments and so forth.

But it would not give me "x-ray vision", since at the end the day it is the same thing I myself conclude if I give the matter 15 seconds of consideration. Something that I miss - a genuine mail written the way a spam might be ("hey! it was nice to meet you!" in the subject, etc), by a recent acquaintance I forgot I made - would be misclassified as spam. There is no x-ray vision involved here - it's not training on the data - it's training on me!

I feel the distinction is an important one.

Re: AI service gives Wikipedians ‘X-ray specs’ to see through bad edits

#20

Post by tptacek »

logicallee wrote: post #19
tptacek wrote: post #14

Earlier quoted context omitted.

That's all the model needs to do to be valuable.

Well, suppose something told me what I would decide if I gave any matter 15 seconds of consideration. It would be an invaluable tool I might use often, for example for sorting and classifying, throwing out trash and so forth. For example, it would be a fantastic spam classifier, since even after 15 seconds there is no doubt. I could use it to get through hundreds of spam messages easily, or delegate things to appropr…

It's 15 seconds of consideration you have to give to huge volumes of edits, so singling out the ones where that 15 seconds is likely to be profitable for an editor is extremely helpful.

(I have no idea how well the scoring works in practice).

Remember, the editorial tasks we're talking about here are enervating.

Post reply on HN