Launch HN: Prolific (YC S19) – Quickly find high-quality survey participants

In my experience running research studies, this was the main problem with MTurk. Things like bots and blatantly junk answers were relatively rare and easy enough to detect and filter. What was much harder to deal with was the fairly high volume of users who just want to finish the survey as fast as possible. It's not so bad for short surveys, but anything over 5 minutes starts to have issues with low quality responses.

We had to introduce several control questions to check for consistency in answers and measure time to find outliers. But, it was not an ideal setup and there were still many survey takers we suspected were not paying much attention. The breakdown for our studies was something like 60% good quality answers, 35% low quality answers that are hard to distinguish from high quality, and 5% total junk answers. We ended up doing an in-person study where we got much cleaner results, presumably because people pay better attention when they feel someone is watching them.

Wages don't seem to do much for this. We were paying a relatively generous $4 per HIT, estimating 30 minutes per HIT when in reality in reality the average time was under 20.

So I'm wondering, are you able to share with us how exactly the "unusual data patterns" and "technical and behavioral checks" can help ensure quality in an ecosystem where users are generally motivated to 1) finish surveys as fast as possible, and 2) appear as if they are giving high quality responses when they are not?

My experience is that most academics pay less than $6.50 per hour for an initial point of contact. For instance, I am currently fielding a survey (N > 5,000 per week, cross-sectional rather than panel) and we pay about $2 all-in, including the provider's charge, for a survey that's about 20 minutes. We'd fall afoul of your compensation rate pretty substantially. If we wanted to do some panel work and we needed re-contact, we'd definitely ramp up our payment quickly to help avoid attrition, but for the first contact, no.

If we wanted to be paying out your rate, we would almost certainly have to get an additional sponsor partner to piggyback some consumer research on top of our actual treatment. We don't want to do this, it's hard enough dealing with our primary funders. This makes me believe you are mostly targeting commercial / market behavior researchers. That's fine, but the pitch suggests you want academics. For transparency's sake, what is your balance of private and university clients as of today?

Second, even working with large sample providers, their pools are often fairly small. We requested 5,000 unique respondents a week for a year and found our sample provider could only guarantee a 1-2 month lockout. Obviously the effective pool you need to guarantee 5,000 * 52 is enormous and so we were expecting to have to negotiate on lockout, but all of this dances around the fact that sample providers are not transparent about the size of their pool and researchers like us are constantly worried about fraud both by sample providers and by respondents. How large is your pool?

Finally, this kind of quota sampling relies on our ability to weight the sample to the population. Weighting is totally permissible, but responsible weighting is going to cap the weight at the high end -- no one wants the one black Republican to skew the entire poll because they have a 150 weight on their observation (this isn't me getting needlessly political, this happened with the USC tracking poll last election cycle). In my experience, the hardest thing about quota sampling as opposed to the old RDD phone samples 20 years ago is that it's very difficult to get high education / high SES / high income respondents. High income respondents should be 10% of the population and they simply are nowhere near 10% of the pools that you get from standard recruitment methods. Can you speak a bit about a) how you recruit high income people into your pool; and b) what percentage of your pool would be high income (say HHI > $125k a year or so).

Finally, what is your pool attrition rate? If someone takes their first Prolific survey today, what is the probability they will still be taking a survey a year from now? It's nice that you have re-contact as part of your system, but the problem in my experience is not figuring out how to recontact, it's getting people to stay engaged for a long time.

Hope you have good answers to these questions, and that if you do, answering them here will help you get positive exposure from other readers.

Right now we’re available to participants in OECD countries and it’s probably going to be another while before we can expand globally. Within the 36 countries that we’re in, we’re currently trying to work out what incentives might work best: How can we encourage the average citizen to sign up and take part in online research? What about very hard-to-reach demographics like professionals?

We feel that cash generally works well as an incentive, but it’s not everything. A lot of the time people want to contribute to projects they personally care about. So on Prolific’s end, it will come down to matching projects with the right participants. Another type of incentive could be to offer to participants that Prolific donates their earnings to charities on their behalf, and perhaps Prolific could match their donation?

We’ve recently launched quota-based representative samples for the US and UK [1], where we stratify based on sex, ethnicity, and age. A by-product of this feature is that it specifically invites niche demographics to participate, helping fill gaps. We hope that having more surveys available for demographics that are in the minority (e.g. certain ethnicities and age groups) will improve our ability to recruit these participants (although there’s a bit of a chicken-and-egg problem here).

Another thing we’d like to do is launch a mobile app for participants, so anyone can take a quick survey on the go (while waiting in line, chilling at home, commuting). Making our surveys more accessible to participants through different channels should help represent more people in society.

And then there’s user experience. We’re working on making our site as self-explanatory as possible, so anybody can sign up and start participating, even if you’re someone who doesn’t spend much time on the internet.

What do you think––what might be other ways to diversify our participant pool? Very curious

[1] https://researcher-help.prolific.co/hc/en-gb/articles/360019...

In addition, SurveyMonkey audience uses a survey exchange provider CINT [1] and as a result you don't have any transparency of where exactly the participants are coming from, how they're vetted, the user experience they're having, and you can't communicate directly with participants or retarget them for longitudinal studies. All of which is important for research and are features that Prolific provide.

[1] https://www.cint.com/press-release/surveymonkey-audience-exp...