Judd Antin
Aaron Shaw
ABSTRACT
In this study we extend research on online collaboration by examining motivation to do work on the crowdsoucing service Amazon Mechanical Turk (MTurk). We address a challenge to many existing studies of motivation in online contexts: they are based on survey self-reports, which are susceptible to effects such as social desirability bias. In addition we investigate a second challenge to the extant research on motivation in the context of MTurk: a failure to examine potential differences between MTurk workers (Turkers) from different parts of the world, especially those from the US and India, MTurk's two largest worker groups. Using a survey technique called the list experiment, we observe distinct profiles of motivation and patterns of social desirability effects among Turkers in the US and India. Among US Turkers, we find that social desirability encourages over-reporting of each of four motivating factors we examined. The over-reporting was particularly large in the case of money as a motivator. In contrast, among Turkers in India we find a more complex pattern of social desirability effects, with workers under-reporting "killing time" and "fun" as motivations, and drastically over-reporting "sense of purpose." We conclude by discussing these results and proposing implications for future research and design.
- S. A. Adams, C. E. Matthews, C. B. Ebbeling, C. G. Moore, J. E. Cunningham, J. Fulton, and J. R. Hebert. The effect of social desirability and social approval on Self-Reports of physical activity. American Journal of Epidemiology, 161(4):389--398, Feb. 2005.Google Scholar
Cross Ref
- Alexa.com. mturk.com - traffic details. http://alexa.com/siteinfo/mturk.com, Dec. 2011.Google Scholar
- M. Buhrmester, T. Kwang, and S. D. Gosling. Amazon's mechanical turk. Perspectives on Psychological Science, 6(1):3--5, Jan. 2011.Google ScholarCross Ref
- D. Chandler and A. Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. University of Chicago mimeo, 2010.Google Scholar
- T. J. DeMaio. Social desirability and survey measurement: A review. In C. F. Turner, E. Martin, and N. R. C. U. P. on Survey Measurement of Subjective Phenomena, editors, Surveying subjective Phenomena, volume 2, pages 257--282. Russell Sage Foundation, June 1984.Google Scholar
- S. Derné. Cultural conceptions of human motivation and their significance for culture theory. In D. Crane, editor, The Sociology of Culture: Emerging Theoretical Perspectives, pages 267--287. Blackwell, Cambridge, MA, 1994.Google Scholar
- D. C. Ganster, H. W. Hennessey, and F. Luthans. Social desirability response effects: Three alternative models. The Academy of Management Journal, 26(2):321--331, June 1983. ArticleType: research-article / Full publication date: Jun., 1983 / Copyright 1983 Academy of Management.Google Scholar
- E. Goffman. The presentation of self in everyday life. Doubleday, NY, NY, 1973.Google Scholar
- E. Gonzalez-Ocantos, C. K. de Jonge, C. Melndez, J. Osorio, and D. W. Nickerson. Vote buying and social desirability bias: Experimental evidence from nicaragua. American Journal of Political Science, Forthcoming.Google Scholar
- P. Ipeirotis. The new demographics of mechanical turk. http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html, Mar. 2010.Google Scholar
- A. L. Janus. The influence of social desirability pressures on expressed immigration attitudes*. Social Science Quarterly, 91(4):928--946, 2010.Google ScholarCross Ref
- N. Kaufmann, D. Veit, and T. Schulze. More than fun and money. worker motivation in crowdsourcing a study on mechanical turk. In Proceedings of the Seventeenth Americas Conference on Information Systems, Detroit, MI, 2011.Google Scholar
- S. Khanna, A. Ratan, J. Davis, and W. Thies. Evaluating and improving the usability of mechanical turk for low-income workers in india. In Proceedings of the First ACM Symposium on Computing for Development, ACM DEV '10, pages 12:1--12:10, New York, NY, USA, 2010. ACM. Google ScholarDigital Library
- S. Kiesler, J. Siegel, and T. W. McGuire. Social psychological aspects of computer-mediated communication. American Psychologist, 39(10):1123--1134, 1984.Google ScholarCross Ref
- S. Kiesler and L. S. Sproull. Response effects in the electronic survey. Public Opinion Quarterly, 50(3):402--413, Sept. 1986.Google ScholarCross Ref
- M. F. King and G. C. Bruner. Social desirability bias: A neglected aspect of validity testing. Psychology and Marketing, 17(2):79--103, Feb. 2000.Google ScholarCross Ref
- A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 453--456, Florence, Italy, 2008. ACM. Google ScholarDigital Library
- J. H. Kuklinski, M. D. Cobb, and M. Gilens. Racial attitudes and the "new south". The Journal of Politics, 59(2):323--349, May 1997.Google ScholarCross Ref
- K. R. Lakhani, R. G. Wolf, J. Feller, B. Fitzgerald, S. Hissam, and K. R. Lakhani. Why hackers do what they do: Understanding motivation and effort in Free/Open source software projects. In Perspectives on Free and open Source Software. MIT Press, Cambridge, Mass, 2005.Google ScholarDigital Library
- H. R. Markus and S. Kitayama. Culture, self, and the reality of the social. Psychological Inquiry, 14:277--283, Oct. 2003.Google ScholarCross Ref
- W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11(2):100--108, 2009. Google ScholarDigital Library
- A. Meade, A. Watson, and C. Kroustalis. Assessing common methods bias in organizational research. In 22nd Annual Meeting of the Society for Industrial and Organizational Psychology, New York, 2007.Google ScholarCross Ref
- K. Mieszkowski. "I make $1.45 a week and i love it". http://www.salon.com/tech/feature/2006/07/24/turks/index_np.html, Sept. 2007.Google Scholar
- K. Mirchandani. Practices of global capital: Gaps, cracks and ironies in transnational call centres in india. Global Networks, 4(4):355--373, 2004.Google ScholarCross Ref
- R. H. Moorman and P. M. Podsakoff. A meta-analytic review and empirical test of the potential confounding effects of social desirability response sets in organizational behavior research. Journal of Occupational and Organizational Psychology, 65(2):131--149, 1992.Google ScholarCross Ref
- B. Nardi, D. J. Schiano, M. Gumbrecht, and L. Swartz. Why we blog. Communications of the ACM, 47(12):41--46, 2004. Google ScholarDigital Library
- A. J. Nederhof. Methods of coping with social desirability bias: A review. European Journal of Social Psychology, 15(3):263--280, 1985.Google ScholarCross Ref
- R. Nisbett and T. Wilson. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(1):231--259, 1977.Google ScholarCross Ref
- O. Nov, M. Naaman, and C. Ye. Motivational, structural and tenure factors that impact online community photo sharing. In Third International Conference on Weblogs and Social Media (ICWSM 2009), San Jose, CA, 2009.Google Scholar
- D. L. Phillips and K. J. Clancy. Some effects of "social desirability" in survey studies. The American Journal of Sociology, 77(5):921--940, Mar. 1972.Google ScholarCross Ref
- A. J. Quinn and B. B. Bederson. Human computation: A survey and taxonomy of a growing field. In Proceedings of ACM CHI 2011 Conference on Human Factors in Computing Systems, 2011. Google ScholarDigital Library
- S. Rafaeli, D. R. Raban, and G. Ravid. How social motivation enhances economic activity and incentives in the google answers knowledge sharing market. International Journal of Knowledge and Learning, 3(1):1--11, 2007.Google ScholarCross Ref
- J. Rogstadius, V. Kostakos, A. Kittur, B. Smus, J. Laredo, and M. Vukovic. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets, 2011.Google Scholar
- J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, pages 2863--2872, Atlanta, Georgia, USA, 2010. ACM. Google ScholarDigital Library
- J. Schroer and G. Hertel. Voluntary engagement in an open Web-Based encyclopedia: Wikipedians and why they do it. Media Psychology, 12(1):96, 2009.Google ScholarCross Ref
- J. Scott. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press, New Haven, CT, 1998.Google Scholar
- A. Shaw, J. Horton, and D. Chen. Designing incentives for inexpert raters. In The 2011 ACM Conference on Computer Supported Cooperative Work (CSCW, 2011), Hangzhou, China, 2011. Google ScholarDigital Library
- S. Weber. The Success of Open Source. Harvard University Press, Apr. 2004. Google ScholarDigital Library
- J. Zittrain. Ubiquitous human computing. Philosophical Transactions of the Royal Society A, 366:3813--3821, 2008.Google ScholarCross Ref
Index Terms
- Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India
Recommendations
A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsThis paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...
Older Adults’ Motivation and Engagement with Diverse Crowdsourcing Citizen Science Tasks
Human-Computer Interaction – INTERACT 2021AbstractIn this exploratory study we evaluated the engagement, performance and preferences of older adults who interacted with different citizen science tasks. Out of 40 projects recently active on the Zooniverse platform we selected top ones to be ...
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsDiversifying participation in crowd work can benefit the worker and requester. Increasing numbers of older adults are online, but little is known about their awareness of or how they engage in mainstream crowd work. Through an online survey with 505 ...
Comments