Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everybody is spamming everybody else on Mechanical Turk

openresearch.wordpress.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Everybody is spamming everybody else on Mechanical Turk

#11

Post by pault »

user24 wrote: post #7
imx wrote: post #3

Earlier quoted context omitted.

Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road... To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different w…

looks like there's a need for a mturk preferred worker aggregation service.

Until someone uses mturk to spam your ratings.

Re: Everybody is spamming everybody else on Mechanical Turk

#12

Post by davidu »

user24 wrote: post #7
imx wrote: post #3

Earlier quoted context omitted.

Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road... To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different w…

looks like there's a need for a mturk preferred worker aggregation service.

it's called crowdflower.

Re: Everybody is spamming everybody else on Mechanical Turk

#13

Post by rms »

scrrr wrote: post #4

It seems like unique online-identities that belong to real people, just like Facebook offers them, seem to be the only way to prevent rating-spam. Or are they? What if "mechanical turks" continue to use their FB-account to do the same? This makes any rating-system almost useless. And since I will be publishing an Android-App soon: Wouldn't it be wise to hire people to rate it with 5 stars, say a few hundred times? It…

Ethics are murky. Don't talk about things like this on public forums. Google's watchful AI is always with us and when it finds you it will crush you. In practice, why hire people to rate it 5 stars when you can pay 200 friends of your friends to download your app and rate it if they like it?

Re: Everybody is spamming everybody else on Mechanical Turk

#14

Post by _delirium »

yummyfajitas wrote: post #10
snikolov wrote: post #9

The article raises an interesting point: that many turkers just assume there is no quality assurance being done on the requester end and everything will automatically be accepted and paid for. Since it is tricky to automate QA for huge sets of tasks I would guess this assumption is mostly correct, and turkers take advantage of it.

What you typically do is include gold data in your dataset. I.e., define a HIT to be 10 sequential tasks, 3-4 of which you know the correct answer to. If a turker can't get the gold data right, their work is rejected.

An even better (and common) strategy is to reject the work that fails the gold-data test and to reject most of the rest of the work too, claiming it failed the gold-data test.

Re: Everybody is spamming everybody else on Mechanical Turk

#16

Post by patio11 »

user24 wrote: post #7
imx wrote: post #3

Earlier quoted context omitted.

Mturk manuals are junk and very hard to follow. HIT data cleansing is the biggest issue. Instead of using the command line tool, use their API to integrate into your app, as it will save plenty of time down the road... To "weed out" ineligible workers, try this approach: 1. Post a bunch (1000-5000) of cheap multiple-choice HITs. 2. Allow no more than 10 hits per worker. 3. Each hit to get 3 responses from different w…

looks like there's a need for a mturk preferred worker aggregation service.

Seriously, for being a fire-and-forget API to the lowest possible level of human tasks, it requires a heck of a lot of hands on management, including arguing with identifiable people over two cents. (YOU DIDN'T SAY TO TURN OFF CAPS. I wish I were exaggerating.)

I ended up writing off five hours to goodwill when I did a project with a $100 turking component for a client. To use a line favored by my old Indian colleagues: if you pay peanuts, you get monkeys. Lesson learned.

Next time I will just find a freelancer with a high tolerance for repetition.

Re: Everybody is spamming everybody else on Mechanical Turk

#17

Post by pavel_lishin »

kkowalczyk wrote: post #5
scrrr wrote: post #4

It seems like unique online-identities that belong to real people, just like Facebook offers them, seem to be the only way to prevent rating-spam. Or are they? What if "mechanical turks" continue to use their FB-account to do the same? This makes any rating-system almost useless. And since I will be publishing an Android-App soon: Wouldn't it be wise to hire people to rate it with 5 stars, say a few hundred times? It…

A solution does exist: providers of mturk-like services could disallow such work items and enforce that (inci-meta-dentally they could use mturk itself to crowd-source spam identification on the cheap). There is additional work for the service provider but it would seem to me that it does align with their self-interest at some level. I don't think Amazon really wants mturk to be associated with providing a spam work…

> providers of mturk-like services could disallow such work items

Except for the one shady site that doesn't, and ends up raking in profits.

Re: Everybody is spamming everybody else on Mechanical Turk

#18

Post by wizard_2 »

davidu wrote: post #12
user24 wrote: post #7

Earlier quoted context omitted.

looks like there's a need for a mturk preferred worker aggregation service.

it's called crowdflower.

These guys constantly prove they know how to manage the kinds of problems Turk and other sources bring to the table.

Re: Everybody is spamming everybody else on Mechanical Turk

#19

Post by jellicle »

snikolov wrote: post #9

The article raises an interesting point: that many turkers just assume there is no quality assurance being done on the requester end and everything will automatically be accepted and paid for. Since it is tricky to automate QA for huge sets of tasks I would guess this assumption is mostly correct, and turkers take advantage of it.

"Tricky to automate" what? Are you not literally in the middle of using a tool that helps you automate QA for huge sets of tasks?

It should be trivial to create a task, create a task for evaluating that task, and yet another task for evaluating that task. Run all three long enough and you will in fact get good results.

Obviously if you're going to use an unreliable protocol there have to be management protocols in effect to correct errors, or you will end up with errors. This is not a revelation.

Post reply on HN