Everybody is spamming everybody else on Mechanical Turk
openresearch.wordpress.com
Everybody is spamming everybody else on Mechanical Turk
1–10 of 29 posts
Re: Everybody is spamming everybody else on Mechanical Turk
#2Eyeopening.
I was planning on using mt for a project I'm working on.
has anyone got any pointers on how to get the best out of mechanical turk? Advice much appreciated.
Re: Everybody is spamming everybody else on Mechanical Turk
#3Eyeopening. I was planning on using mt for a project I'm working on. has anyone got any pointers on how to get the best out of mechanical turk? Advice much appreciated.
Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road...
To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different workers. 4. Review answers, compile the list of "good" workers, blacklist the "bad" ones. 5. Post another bunch of HITS, make them available for eligible workers only (found in step 4), this time the HITs might be more demanding, individually review results for each worker -> the best ones go on your "preferred worker" list. 6. Repeat steps 1-5 as necessary.
From then on it's fairly safe to rely on mturk workers from your preferred list.
Re: Everybody is spamming everybody else on Mechanical Turk
#4It seems like unique online-identities that belong to real people, just like Facebook offers them, seem to be the only way to prevent rating-spam.
Or are they?
What if "mechanical turks" continue to use their FB-account to do the same?
This makes any rating-system almost useless.
And since I will be publishing an Android-App soon: Wouldn't it be wise to hire people to rate it with 5 stars, say a few hundred times? It seems like my competition will do it.
Re: Everybody is spamming everybody else on Mechanical Turk
#5It seems like unique online-identities that belong to real people, just like Facebook offers them, seem to be the only way to prevent rating-spam. Or are they? What if "mechanical turks" continue to use their FB-account to do the same? This makes any rating-system almost useless. And since I will be publishing an Android-App soon: Wouldn't it be wise to hire people to rate it with 5 stars, say a few hundred times? It…
A solution does exist: providers of mturk-like services could disallow such work items and enforce that (inci-meta-dentally they could use mturk itself to crowd-source spam identification on the cheap).
There is additional work for the service provider but it would seem to me that it does align with their self-interest at some level. I don't think Amazon really wants mturk to be associated with providing a spam work force.
I believe one of the things that CrowdFlower explicitly calls out as an advantage over mturk is quality control (although for this particular solution to work all crowd-sourcing providers would have to do it - in this particular case it takes only one bad provider to enable bad behavior.
As to your hopefully hypothetical question: a risk you're running is that Google will pull out your app from the store. I haven't heard a case with Google but I'm pretty sure apps were pulled from Apple's App Store for manipulating ratings, so the downside could be big (your hard work could amount to nothing).
Re: Everybody is spamming everybody else on Mechanical Turk
#6Money quote:
> We all know that Mechanical Turk challenges the whole “Junk-in, Junk-out” dilemma and makes it more like “Always junk-out, regardless of the input process”
Couldn't be more true IMO, mturk is basically useless except for this "meta" kind of research and is a good example of a community that needed active management and positive incentives going to absolute shit in the absence of both.
Re: Everybody is spamming everybody else on Mechanical Turk
#7Eyeopening. I was planning on using mt for a project I'm working on. has anyone got any pointers on how to get the best out of mechanical turk? Advice much appreciated.
Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road... To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different w…
looks like there's a need for a mturk preferred worker aggregation service.
Re: Everybody is spamming everybody else on Mechanical Turk
#8I've spent a good chunk of time modifying an image labeling interface to make it more intuitive for mechanical turk workers to label obscure things like biological scans. The hope is that a better interface will increase quality (they did not seem to have a clue what to do using the old interface), but I'm starting to question whether the interface will make that much of a difference after all.
Re: Everybody is spamming everybody else on Mechanical Turk
#9The article raises an interesting point: that many turkers just assume there is no quality assurance being done on the requester end and everything will automatically be accepted and paid for. Since it is tricky to automate QA for huge sets of tasks I would guess this assumption is mostly correct, and turkers take advantage of it.
Re: Everybody is spamming everybody else on Mechanical Turk
#10The article raises an interesting point: that many turkers just assume there is no quality assurance being done on the requester end and everything will automatically be accepted and paid for. Since it is tricky to automate QA for huge sets of tasks I would guess this assumption is mostly correct, and turkers take advantage of it.
What you typically do is include gold data in your dataset. I.e., define a HIT to be 10 sequential tasks, 3-4 of which you know the correct answer to. If a turker can't get the gold data right, their work is rejected.