What is MTurk?
MTurk is a website run by Amazon that works as a readily available marketplace to match “workers” with available work from various “requesters.” Amazon describes MTurk as, “a marketplace for work that requires human intelligence. The Mechanical Turk service gives businesses access to a diverse, on-demand, scalable workforce and gives workers a selection of thousands of tasks to complete whenever it’s convenient.”
How are academic researchers using MTurk?
MTurk allows researchers to have access to a large population of willing participants for research studies. Researchers can generate a HIT (Human Intelligence Task) that gives MTurk users a Title and description of the online task. It also states the amount of compensation users will receive and the amount of time it takes to complete. Because MTurk is being used primarily as a recruitment site, researchers have been routing users to links for their online survey’s that are housed at different online survey software sites (i.e. – Qualtrics, Survey Monkey, Google Forms, Survey Gizmo, etc.). MTurk houses a more diverse, heterogeneous, population that is more attractive to researchers than the typical college campus population. It also allows for pseudo-anonymity between researchers and participants by having a third party (Amazon) overseeing payments (compensation) to participants.
Amazon Mechanical Turk is a site primarily for matching people with work and submitting payments to workers. It is not designed with an eye towards human subjects research. It lacks the sophistication and security measures of dedicated survey software tools and collection of data by Amazon is subject to Amazon’s Privacy Policy and Terms of Service.
What do academic researchers need to consider when submitting an MTurk study to the IRB?
Recruitment: The title of the study and the description of the HIT are acting as a form of recruitment. Researchers should be sure to include the title and HIT description as part of the Kuali application. In the description, researchers should be sure to include the following information:
- Researchers should be clear about compensation and bonuses. Also, it would be useful for participants to know how long it will take for the researchers to approve their HIT (i.e. – how long it takes for turkers to receive payment).
- Clearly and accurately state the time required to complete the task.
- Participants should be told if there is a screener in order to qualify. It would be important for researchers to make clear if participants are being paid for the time it takes to complete the screener or not. One option would be to list qualifications for participation in the description. Another option would be to make a hit for the Screener (which would pay a nominal amount) and then if a turker is eligible for the main study, they get a bonus amount or are invited to a follow-up HIT which includes the main study.
- If any extraneous software is required to complete the task, this should be stated in the description (e.g. this task requires javascript or inquisit).
- Researchers should be clear about the type of task participants are being asked to do. For instance, if the task involves writing, or watching videos, this should be stated in the description. Also be aware that certain types of tasks, such as writing tasks, elicit higher compensation.
- The researchers name and/or school affiliation should be listed either as the Requester or in the description of the HIT.
- If applicable: The link to the online survey should be included.
Consent: The first page of the online survey should be the consent document. The online consent will have all of the elements of a regular consent, but it will not require a signature. Participants will either click an “I Agree” or an “I do not Agree” box. The “I Agree” box will take them into the survey. The “I do not agree” box will thank them for their time and take them away from the survey. For a sample of an online consent form, please see our Informed Consent Templates forler.
Debrief: If the researchers are using deception or incomplete disclosure (i.e. – are not stating exactly what the study is about so as not to bias participants responses), then it is important to include a debriefing form at the end of the survey. This debriefing form could be embedded into the last page of the survey and would require participants to answer a final question allowing researchers to use their data (or not use their data) now that they know the true purpose of the study. For more information on debriefing forms, please see our debriefing process. Please note: For particularly sensitive topics, the IRB may want to ensure that participants receive a debriefing form, even if they do not complete the full study (i.e. – they click out before the end of the survey). This might mean contacting participants (through MTurk) and providing them a debriefing form. If this is something that will happen for your study, it would be important to include a statement in the HIT description and in the informed consent that the researchers will be providing participants with additional information after the study and that they may contact participants through MTurk. This might mean collecting participants MTurk worker IDs. Researchers should be sure to state that the collection of MTurk worker IDs will only be for debriefing purposes and after debriefing, MTurk worker IDs will be deleted and will at no time ever be linked to their survey data.
Confidentiality: While it may have been the goal that MTurk workers were anonymous to academic researchers, the reality is that anonymity cannot be guaranteed in any online environment were data is being collected. Recent research shows that MTurk worker IDs can easily be linked to individuals Amazon profiles including individuals wish lists and previous product reviews. This means that researchers must be careful in deciding what information to collect from participants. The default should be that participants MTurk worker IDs not be collected. If it is necessary to collect worker IDs, then the researchers should ensure that worker IDs are kept confidential and secure, are not linked back to survey data, and are deleted after use.
Additional things to consider with regard to confidentiality of participants identity and data are the Amazon MTurk (AMT) Terms of Service (TOS) and online survey software (qualtrics, survey monkey, google forms, survey gizmo, zoho survey, etc.). TOS especially as it relates to the collection of participants online behavior and history (through the use of cookies or other tracking systems) and the selling of participants data to third parties. It would also be important for researchers to be aware of any policies or procedures in place by AMT and/or online survey software companies when a breach of data occurs. It might be helpful to think about the researchers own protocol for when a breach of data security occurs. Working with IT to solidify this process may be necessary. This information could then be relayed to participants through the consent.