Tips and Recommendations:
As a researcher, it can sometimes be challenging to develop sampling methods in the digital age. As a research team, SMART Labs has tried various online sampling strategies to render quality data. We have developed this Tips and Recommendations page specifically for researchers to hopefully serve as a useful resource when deciding on which online sampling method to use: Amazon Mechanical Turk, Facebook Ads or Qualtrics Panels. We have also included a brief review on the use of electronic gift cards as participant incentives. This overview draws on our own team's experience with the online methods described below and does not mean to promote or discourage the use of any or all of the methods mentioned on this page.
Amazon Mechanical Turk
"Amazon Mechanical Turk (MTurk) is a marketplace for work that requires human intelligence. The MTurk service gives businesses access to a diverse, on-demand, scalable workforce and gives Workers a selection of thousands of tasks to complete whenever it's convenient.
MTurk is based on the idea that there are still many things that human beings can do much more effectively than computers, such as identifying objects in a photo or video performing data de-duplication, transcribing audio records, or researching data details. Traditionally, tasks like this have been accomplished by hiring a large temporary workforce (which is time consuming, expensive, and difficult to scale) or have gone undone."
Researchers are attracted to the service for several reasons... compared to traditional samples recruited through universities or panels, the Mturk service is timely, inexpensive, and targeted. Academics can either choose to dissect their studies and create smaller Human Intelligence Tasks (HITs) or they can post full surveys for Workers to complete in an expedited fashion. Although MTurk has proven to be a efficient and cheap alternative to expensive panel sampling, and just as qualified as university samples, there are some things to keep in mind about the service:
(1) Money does not work as a big moderator for data quality--setting qualifying criteria does.
(2) To avoid the haunting 40 percent Mturk fee, and as best practice, consider running at least 1 soft launch (i.e. 9 or under assignments at a time) anything 10 or over incurs the 40 percent fee rather than the standard 20 percent fee.
(3) Roughly a 50 percent retention rate for longitudinal studies
(4) For cross-sectional and longitudinal studies pay the same each round, no need to increase.
(5) Workers are hyper-sensitive to attention checks. Set timers and force responses whenever it makes sense to do so.
(6) Setting qualifications are important, but you are going to get people outside the USA whether you like it or not due to falsified geo-locations. As a good rule of thumb, you should set your qualifications to 95 percent approved hits, over 1000 approved hits, and location is United States.
(7) It is thought that 97% of workers are overworked.
Some other things to keep in mind when considering MTurk as a sampling alternative is that it is good practice to balance with student sample to put down nay-sayers that will likely surface in the journal review stages. However, some reviewers frown upon split samples--so weigh your options and do what's best for your study. Keep an eye on MTurk Forums--people will likely talk about your study. Check up on them and consider legitimate concerns. If using Qualtrics as your landing site, it works well with the Mturk but you must include a unique ID code for payment. For researchers interested in viewing blogs dedicated to Mturk, visit A Computer Scientist in a Business School and Expirimental Turk.
For those of you wondering about a monetary average, researchers commented in an open SPSP forum and decided on minimum, mid-range and maximum payment averages. The average pay was $0.1379 per minute (SD = 0.059). The max pay mentioned was $0.25 per minute ($15 per hour) and the minimum was $0.05 per minute ($.50 for 10 minutes). If you'd like to read more on this forum topic, as a member of SPSP, you can read the thread here.
Qualtrics Panels has many service options to choose from. Our team decided to use the Purchase Respondents service which connects your research team with a team of Qualtrics professionals: usually including a sales person who compiled a quote and a project manager who collects and delivers your data. With this service, as a researcher, you prepare the survey on your own and the Qualtrics folks find you qualifying respondents so we could collect data without our having to purchase/collate our own panel. As a researcher, you can target a general sampling of the United States population, or a general sampling of select other countries. You can also target a specific age range, gender, US state or ethnicity.
With the Purchase Respondents service, Qualtrics (the sample experts) promises to find you respondents on-demand, launch your survey, and monitor your project. Qualtrics boasts high-quallity sample, a convenient launch process, and thousands of projects per year.
In our experience with the panel provider, Qualtrics does not always provide quality respondents. In fact, our data was of better overall quality with MTurk Workers than it was with Qualtrics Panel respondents. Something to consider if you, like us, are routinely attempting to access internet users and online daters in particular. However, considering the fact that we target a specific, media literate audience, one could argue that MTurkers are inherently more media literate than the average panelist.
"Facebook advertising is aimed at businesses, large, medium or small, as a growth tool. To get started as a business, you choose your objective; select your audience based on demographics including, but not limited to, age, location or interests; decide where to run your ad (i.e., Facebook, Instagram or the apps and websites in Audience Network); set your budget (either daily or lifetime); pick a format (i.e., single or multiple images, videos, slideshow or Canvas ad); place your order/enter your payment details; measure and manage ad effectiveness."
In our team's experience, Facebook ads were extremely inefficient as a recruitment method for our sample. We had a very low response rate coupled with a very high attrition rate. To put it simply, one participant cost us 68 valuable advertising dollars. This method might work best for brand awareness rather than purposive sampling. Creating Google ads with targeted demographics might be a better use of your valuable ad dollars.
Electronic Gift Cards
Track purchases, disbursements and recall history. Must always have a paper trail--keep emails, receipts, etc. Include IRB language to recall unused gift cards a generous time period with a finite amount of follow up reminders.
Get Elizabeth’s amended approval from her last study to include language: i.e., "After two weeks of issue, if the gift card has not been redeemed, the researcher will reclaim and re-purpose the funds for new participants"
For e-gift cards, it is good to have an official, team or lab account rather than a personal account. This helps to establish professionalism and legitimacy in case of an audit. Funds that come back to you can be recycled for a later study.
Do NOT overbuy. Rather, buy as you go.
Can also consider doing a draw to simplify the process (i.e., 1 in 50 chance to win a 200 dollar gift card).
Please note, if an employee of a university employee is the recipient of a gift card, most institutions expect department administrators to notify payroll of "additional income" for the employee. In other words, researchers are expected to report gift cards to employees as taxable earnings to payroll. This seems silly to us, but it's a policy we must comply with.