Mechanical Turk: Now with 40.92% Spam.
behind-the-enemy-lines.blogspot.com
Mechanical Turk: Now with 40.92% Spam.
1–10 of 28 posts
Re: Mechanical Turk: Now with 40.92% Spam.
#2Fascinating.
My experience with MTurk suggests that Amazon has mostly abandoned the product. Amazon still uses it for product classification, but does not participate in the community forums (even on the official AWS MTurk developer forum). No new features have appeared in several years, and enhancement requests have fallen on deaf ears.
That said, MTurk still works. The API still does what its documentation claims. People still perform tasks. Payment still works.
Re: Mechanical Turk: Now with 40.92% Spam.
#3Fascinating. My experience with MTurk suggests that Amazon has mostly abandoned the product. Amazon still uses it for product classification, but does not participate in the community forums (even on the official AWS MTurk developer forum). No new features have appeared in several years, and enhancement requests have fallen on deaf ears. That said, MTurk still works. The API still does what its documentation claims.…
I have to say that although it feels that way, lately they've been sort of active.
They've been using twitter since October at: http://twitter.com/amazonmturk
And they just launched launched a blog at: http://mechanicalturk.typepad.com/
Largely I attribute this to Natala Menezes, their newish Product Manager.
Re: Mechanical Turk: Now with 40.92% Spam.
#4At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app).
Then I tried a new tactic. I hired some pros from various translation websites. All were more expensive (money-wise) than MT, but the results were better, and I spent a lot less time. Some were even close to MTurk prices.
So, I don't use MTurk anymore. Maybe if I had a more mindless task I'd try again, but it is a lot of hassle.
I know this isn't exactly what the author means by "spam" - but even the workers on that site are often dishonest. It's a total black market.
Re: Mechanical Turk: Now with 40.92% Spam.
#5It's unbelievable that no one has swooped in to crush MTurk given how little Amazon has done to develop it since 2005(!)
Re: Mechanical Turk: Now with 40.92% Spam.
#6At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app). Then I tried a new tactic. I hired some pros from various translation…
It seems to be a failed social experiment to me, there is definitely a negative feedback loop there. Bad HITS, low wages, withheld payments, bad workers, sweatshops. Hard to imagine it will get any better.
I wonder if you reversed the process and had projects bid on workers if it could counter the race to the bottom effect. Include a system where work could be divided into common types and include worker metrics and feedback that captured any specialization. Perhaps allow some workers to fill a managerial role, where they acquire large blocks and have a trusted group they farm out work to. Presumably wages would go up but bad work would go down, perhaps offsetting it somewhat.
Re: Mechanical Turk: Now with 40.92% Spam.
#7It's unbelievable that no one has swooped in to crush MTurk given how little Amazon has done to develop it since 2005(!)
I think this is one of those examples where nobody's rushing in because no matter how you slice it, the product is just not that useful.
I've used it a few times and because the eco system is truly about hiring dirt, dirt cheap labor you kind of get what you pay for.
Beyond that the revenue model sucks - the concept of running millions of micro-transactions and having a little piece of all the action is compelling, but again because the product use is so limited it's difficult to make signifigant revenue.
Re: Mechanical Turk: Now with 40.92% Spam.
#8At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app). Then I tried a new tactic. I hired some pros from various translation…
I had good success stopping cheating by using images of text instead of actual text. Typing in a foreign language is not trivial if it's not something you're used to doing.
I used a python wrapper for pango-view to generate the text. I have a python module for this here: https://github.com/j2labs/pyango_view
Nothing complicated about what I'm doing but the github page shows how nice the text can really look.
Re: Mechanical Turk: Now with 40.92% Spam.
#9At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app). Then I tried a new tactic. I hired some pros from various translation…
I had good success stopping cheating by using images of text instead of actual text. Typing in a foreign language is not trivial if it's not something you're used to doing. I used a python wrapper for pango-view to generate the text. I have a python module for this here: https://github.com/j2labs/pyango_view Nothing complicated about what I'm doing but the github page shows how nice the text can really look.
It might also be of interest that I recently posted a gist that shows how to post some basic mechanical turk tasks using Python's boto library.
Re: Mechanical Turk: Now with 40.92% Spam.
#10At first, I thought MTurk was good for getting translation work done, and checked by others. It worked OK, but I had to be clever to avoid the cheaters who would ignore instructions not to use machine translation. After refining my tactics, talking to a lot of workers, and fending off the crooks, I got a few decent translations (for my iPhone app). Then I tried a new tactic. I hired some pros from various translation…
Not 100% relevant to this thread, but if it helps, I've had success with http://www.icanlocalize.com for localizing apps. They assign a human translator for your strings and are accommodating with custom directions you might have. They'll even take your strings file from Xcode.
(Disclaimer: I'm not affiliated with this site, your mileage may wary.)