Case Study: Accuracy & Fraud on MTurk
tumblr.gethumanoid.com
Case Study: Accuracy & Fraud on MTurk
1–9 of 9 posts
Re: Case Study: Accuracy & Fraud on MTurk
#2"Humanoid’s software identified 8% of the workforce as engaged in fraud." <--that is a big number!
Re: Case Study: Accuracy & Fraud on MTurk
#3Interesting story -- I've attempted to use mturk a few times in the past, but it is actually a lot of work to get it right. Good instructions are really important, as well as a way to measure performance.
Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."
Re: Case Study: Accuracy & Fraud on MTurk
#4Interesting story -- I've attempted to use mturk a few times in the past, but it is actually a lot of work to get it right. Good instructions are really important, as well as a way to measure performance. Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."
good instructions are crucial.
Re: Case Study: Accuracy & Fraud on MTurk
#5Great data, very cool how sophisticated algorithms can increase performance dramatically for human-powere tasks
Re: Case Study: Accuracy & Fraud on MTurk
#6Fails to mention how much they cost on top of the mturk prices to the client
Re: Case Study: Accuracy & Fraud on MTurk
#7What does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?
Re: Case Study: Accuracy & Fraud on MTurk
#8I just used Humanoid for the first time. I had a job that would have taken about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog URL, and their CEO's Twitter account and blog URL).
Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent.
P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop.
(Disclosure: I am an investor. But I'm giving honest feedback.)
Re: Case Study: Accuracy & Fraud on MTurk
#9What does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?
Junk results, mostly, yes. That way they can complete more taks faster and still make money because most of the time their work is not manually approved.