Interesting article, but probably poorly timed submissing to HN. You could have used this article to promote your product, but you're in stealth mode so we get a "sign up to hear when my product is available" page, but still have no idea what your product is. Good article, bad marketing. Should have released this about a week after you launched the site :)
How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
51–60 of 60 posts
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#52Earlier quoted context omitted.
Yeah, I've done this before. Of the 200 people I surveyed a much higher % (around 95%) said they liked and would use the service...and then did not. My experience with this stuff makes me think that people who do it generally have a perception that you are their "boss" and that they need to sugar coat everything they tell you. TLDR: Be HIGHLY suspect of any results you get from mechanical turk (same goes for feedback…
It applies to everyone. Not just mechanical turk folks. When someone says they 'will use' your product they are non-committal. (I have done this so many times for the fear of offending the person who is asking me). When its time to take out their credit cards then they backtrack. They are 2 different decision points. The later is what matters to startups.
When its time to take out their credit cards then they backtrack
And to be fair, even if I am trying to be super-honest, I find it very hard to decide whether I would be happy to whip out a credit card from a concept pitch. It's not just a question of "would I pay for A", it is a question of "would I pay for A it it really rang my bell", and I can't tell whether you are going to ring my bell or not until I have tried your product and your customer support philosophy.
When I look at my credit card statement, I see a bunch of companies not just whose product I like and use, but also a bunch of folks for whom I can say "I like the way they do business".
I didn't pay to cloud-store photos for over a decade, until I found smugmug. So what is the right answer to the question "would you pay to store photos"?
This isn't the kind of nuance you would get from an MT survey I don't think.
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#53This is Lindsey - who wrote the post. I wanted to clarify a couple of things, based on the feedback I’m seeing here. (btw. This was my first blog post :)). I am still in stealth mode (which is a completely different conversation that I’ll be blogging about soon). www.Swayable.com will be a consumer facing free service. With that said, the Turk system will not work for a lot of startups that need niche specific data.…
You are in stealth mode, yet willing to pitch the idea to hundreds of random strangers?
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#54I wonder how many of these 200 people were from India, which appears to be supplying a good chunk of Mech Turk userbase.
I also ran a couple of polls with similar intent on MT and then stopped doing that because results were skewed dramatically. For one there's no filtering by a geographical region and you bet your 10c-per-answer that you will get a ton of replies from the poorest corners of the world. Secondly, responders are afraid of you rejecting their answer and thus affecting their MT rating, so they tend to tell you want you want to hear and not what they actually might be thinking. Hence your 73% approval rate.
This sampling is not random, very far from it. You are effectively sampling MT community with its own dynamics, the community that is likely NOT to be representative of your own target user base.
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#55In case anyone doesn't know what a Mechanical Turk is: http://en.wikipedia.org/wiki/The_Turk
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#56I wonder how many of these 200 people were from India, which appears to be supplying a good chunk of Mech Turk userbase. I also ran a couple of polls with similar intent on MT and then stopped doing that because results were skewed dramatically. For one there's no filtering by a geographical region and you bet your 10c-per-answer that you will get a ton of replies from the poorest corners of the world. Secondly, resp…
I've been using CrowdFlower as an interface to MTurk, and it let me exclude specific countries. Your point about the secondary motivation is unfortunately valid, though.
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#57Earlier quoted context omitted.
It is still waaaaaaaaaaay better than asking your family and friends.
Dude. No. They're both 100% useless. Mechanical Turkers are incentivized to give positive feedback because there's a chance (lurking in the back of their minds, surely) that people are emotionally sensitive and if they get criticized, they will be emotionally wounded and won't approve their HIT. Do you know how I know this? I asked Turkers to give me feedback on one of my sites, and it was nothing but sunshine and ra…
I guess you could ignore the 73% that said they would use it. But note that 27% said they wouldn't use it and also stated "WHY they wouldn’t use" it. This should be at least 1% useful.
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#58Earlier quoted context omitted.
I've run an MTurk survey as well and asked standard demographic information. My results are surprisingly in line with national averages. I ran the survey at the suggestion of someone with much more experience running MTurk surveys, who claims he has verified that the results are better than typical phone and mall campaigns.
>I've run an MTurk survey as well and asked standard demographic information. My results are surprisingly in line with national averages. Do MTurkers tell the truth though? Presumably one person can run a dozen fronts so that they can answer questions that require a specific demographic. Or they masquerade as elderly females to appear more trustworthy and shame you out of objecting if they've not done the job properl…
Do you think they would lie so as to mirror the national averages?
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#59For anyone else interested in this "quick, cheap, feedback" cycle, PickFu is another service that does this: http://pickfu.com/ I spent about $20 to get 200 responses for my question, but only ended up collecting 103 answers before the question expired. None of this stuff is perfect, but employing a few of these approaches all at the same time will generate results that can be handy to correlate.
Actually, it didn't expire, it finished with 104 of 200 answers picking "B - Paypal" http://pickfu.com/00PULC Also, just to clarify, it's $17 for 200 responses. Thanks for trying out PickFu!
Justin, my apologies on that, I misread the screen and thought I only got 104 answers.
It was a great experience overall, and for micro-testing (where something like Usability testing is too much) PickFu is on my short list for future projects.
Re: How I Used Amazon’s Mechanical Turk to Validate my Startup Idea
#60Earlier quoted context omitted.
Actually, it didn't expire, it finished with 104 of 200 answers picking "B - Paypal" http://pickfu.com/00PULC Also, just to clarify, it's $17 for 200 responses. Thanks for trying out PickFu!
Justin, my apologies on that, I misread the screen and thought I only got 104 answers. It was a great experience overall, and for micro-testing (where something like Usability testing is too much) PickFu is on my short list for future projects.
No worries. Other people have misread it too so we probably need to tweak the wording to make it more clear.
Glad to hear it!