31–40 of 60 posts
Does Amazon say anywhere how diverse Turk users are? Maybe you ended up only surveying mostly one gender in mostly one part of the world?
We did a blog post referencing a study:
http://blog.markiter.com/#turk
http://behind-the-enemy-lines.blogspot.com/2008/03/mechanica...
Which has been updated recently. Some high quality data presented in a clear way.
http://behind-the-enemy-lines.blogspot.com/2010/03/new-demog...
Force them to make a choice. Present Web app A (real), Web app B (dummy), and Web app C (dummy). Make them rank which web app is best.
Set up the survey three ways: A, B, C. B, C, A. and C, B, A. Have a third of the sample take each survey. You would be surprised at the first choice bias with Mechanical Turk. Actually you wouldn't be surprised when you remember that these people just want to get the thing done.
Finally a good secondary survey is to make them rank order features for their worth. This helps find your MVP. (Read more about "conjoint analysis" if this interests you.)
Earlier quoted context omitted.
It is still waaaaaaaaaaay better than asking your family and friends.
Dude. No. They're both 100% useless. Mechanical Turkers are incentivized to give positive feedback because there's a chance (lurking in the back of their minds, surely) that people are emotionally sensitive and if they get criticized, they will be emotionally wounded and won't approve their HIT. Do you know how I know this? I asked Turkers to give me feedback on one of my sites, and it was nothing but sunshine and ra…
You could compare a few pictures, videos, landing pages, etc. And get some a vote with a comment on which is better.
Forced more legwork on the user to actually create multiple options.
For anyone else interested in this "quick, cheap, feedback" cycle, PickFu is another service that does this: http://pickfu.com/ I spent about $20 to get 200 responses for my question, but only ended up collecting 103 answers before the question expired. None of this stuff is perfect, but employing a few of these approaches all at the same time will generate results that can be handy to correlate.
Also, just to clarify, it's $17 for 200 responses.
Thanks for trying out PickFu!
People who use AMT for earning money may not necessarily be the right audience you are looking for.
It costs you nothing to say "yes", but everything changes as soon as you start to charge for it.
Going from Free to $0.01 for any online product or service significantly changes your conversion numbers.
You might get better feedback standing on a corner downtown, asking if people would use your service... and charge them $1 for a voucher that would allow them to use it when it is ready.
PS: Please get out of stealth mode!
It's not clear from the article what kind of business model the Startup in question is planning to use. This is relevant, as the fact the 73% of random strangers said they would "use the service as described" doesn't give any indication that they'd be willing to pay for it. Clearly, the author got his $27.50 worth; how much that's really worth, though, in the long term, remains to be seen.
Yeah, I've done this before. Of the 200 people I surveyed a much higher % (around 95%) said they liked and would use the service...and then did not. My experience with this stuff makes me think that people who do it generally have a perception that you are their "boss" and that they need to sugar coat everything they tell you. TLDR: Be HIGHLY suspect of any results you get from mechanical turk (same goes for feedback…
Does Amazon say anywhere how diverse Turk users are? Maybe you ended up only surveying mostly one gender in mostly one part of the world?
When we were creating http://markiter.com We did a blog post referencing a study: http://blog.markiter.com/#turk http://behind-the-enemy-lines.blogspot.com/2008/03/mechanica... Which has been updated recently. Some high quality data presented in a clear way. http://behind-the-enemy-lines.blogspot.com/2010/03/new-demog...
http://behind-the-enemy-lines.blogspot.com/2010/03/new-demog...
* United States: 46.80% (65% female)
* India: 34.00% (70% male)
* Miscellaneous: 19.20%