Claus Bossen
Christian Dindler
Ole Sejer Iversen
ABSTRACT
This paper focuses on evaluation in Participatory Design (PD), and especially upon how the central aims of mutual learning, empowerment, democracy and workplace quality have been assessed. We surveyed all Participatory Design Conference papers (1990-2014) and papers from special journal issues on PD, focusing on systematic, explicit evaluations. The survey resulted in 143 papers of which 66 were deemed relevant. Of these, 17 papers deal with evaluation of the above mentioned aims. Based on evaluation theory, we propose seven key questions through which to characterize evaluations in PD and analyze the 17 papers. Our analysis reveals that formal evaluations of PD's aims are rare; generally lack details on methods; are researcher- and not participant-led, and that a corpus of work around evaluations needs to be developed. We suggest more explicit, systematic evaluations of PD's central aims to enhance accountability, learning and knowledge building, and to strengthen PD internally and externally.
- Beck, E., Obrist, M, Bernhaupt, R. and Tscheligi, M. 2008. Instant card technique: how and why to apply in user-centered design. In Proc. of PDC '08. IN, USA, p. 162--165. Google Scholar
Digital Library
- van den Besselaar, P. 1998. Democracy & technological change: Limits to steering. In Proc. of PDC'98, Seattle, WA. USA, p. 1--10.Google Scholar
- Reich, B. H., Gemino, A. and Sauer, C. 2008. Modeling the knowledge perspective of IT projects. Project Management Journal, 39(S1), S4-S14. DO=10.1002/pmj.20056Google ScholarCross Ref
- Bossen, C., Dindler, C. and Iversen, O. S. 2010. User gains and PD aims: assessment from a participatory design project. In Proc. of PDC '10. ACM, New York, NY, USA, p. 141--150. DOI=http://dx.doi.org/10.1145/1900441.1900461 Google ScholarDigital Library
- Bossen, C., Dindler, C. and Iversen, O. S. 2012. Impediments to user gains: experiences from a critical participatory design project. In Proc. Of PDC'12, Vol. 1. ACM, New York, NY, USA, p. 31--40. DOI=http://dx.doi.org/10.1145/2347635.2347641 Google ScholarDigital Library
- Braa, K. 1992. Influencing System Quality by Using Decision Diaries in Prototyping Projects, in Proc. of PDC'92, Palo Alto, CA. USA p. 163--170.Google Scholar
- Brandon, P. R. and Ah Sam, A. L. 2014. Program Evaluation. In: The Oxford Handbook of Qualitative Research, Patricia Leavy (ed), p. 471--497.Google Scholar
- Bratteteig, T. and Wagner, I. 2014. Design decisions and the sharing of power in PD. In Proc. of PDC '14, Vol. 2. ACM, New York, NY, USA, p. 29--32. DOI=http://dx.doi.org/10.1145/2662155.2662192 Google ScholarDigital Library
- Clement, A., Griffiths, M. and van den Besselaar, P. 1992. Participatory Design Projects: A retrospective Look. Proc. of PDC'92, Palo Alto, CA. USA, 81--89.Google Scholar
- Costantino, T. LeMay, S. Vizard, L., Moore, H., Renton, D. Gornall, S. and Strang, I. 2014. Participatory design of public library e-services. Proc. of PDC'14, Windhoek, Namibia, p. 133--136. DOI=http://dx.doi.org/10.1145/2662155.2662232 Google ScholarDigital Library
- Dahler-Larsen, P. 2011, The Evaluation Society. Stanford, Stanford University Press, 2011.Google Scholar
- Ehn, P. 1990. Work-Oriented Design of Computer Artifacts. L. Erlbaum Assoc. Inc., Hillsdale, NJ, USA. Google ScholarDigital Library
- Frauenberger, C. Good, J. Fitzpatrick, G. and Iversen, O. S. 2015. In pursuit of rigour and accountability in participatory design. In: International Journal of Human Computer Studies, 74, p. 93--108. Google ScholarDigital Library
- Garde, J. A. and van der Voort, M. C. 2014. Participants' view on personal gains and PD process. In Proc. of PDC '14, Vol. 2. ACM, New York, NY, USA, p. 79--82. DOI=http://dx.doi.org/10.1145/2662155.2662194 Google ScholarDigital Library
- Gerrard, V. and Sosa, R. 2014. Examining participation. In Proc. of PDC '14, Vol. 1. ACM, New York, NY, USA, p. 111--120. DOI=http://dx.doi.org/10.1145/2661435.2661451 Google ScholarDigital Library
- Greene, J. C. 2004. Evaluation, Democracy and Social Change. In The Sage Handbook of Evaluation, Ian F. Shaw, Jennifer C. Greene & Melvin M Mark (eds), p. 118--140.Google Scholar
- Guba, E. G. and Lincoln, Y. S. 1989 "Fourth Generation Evaluation", Newbury Park, Sage.Google Scholar
- Hallberg, N., Pilemalm, S., Jägare, D., Irestig, M. and Timpka, T. 2000. Quality function Deployment (QFD) extended Future Workshop: An Approach for Effective and Enjoyable User Participation, Proc. Of. PDC'00, New York, New York, USA, p. 86--95.Google Scholar
- Hemmings, T., Clarke, K. Rouncefield, M., Crabtree, A. Rodden, T. 2002. Probing the Probes, in Proc. of PDC'02, Malmö, Sweden, p. 42--50.Google Scholar
- Hertzum, M. and Simonsen, J. 2010. Effects-driven IT development: an instrument for supporting sustained participatory design. In Proc. of PDC '10. ACM, New York, NY, USA, p. 61--70. DOI=http://dx.doi.org/10.1145/1900441.1900451 Google ScholarDigital Library
- Hirschheim, R. A. 1983. Assessing participative systems design: Some conclusions from an exploratory study. Information & Management. 1983;6(6):317--27. DOI=http://dx.doi.org/10.1016/0378-7206(83)90040-XGoogle Scholar
- Hornecker, E. Eden, H. and Scharff, E. 2002. In MY situation I would dislike THAAAT!--- Role Play as Assessment Method for Tools Supporting Participatory Planning, in Proc. of PDC'02, Malmö, Sweden, p. 243--247.Google Scholar
- Hänninen, R. and Blom, J. 2012. Every breath you take: use of sensitizing methods in the design of air quality services. In Proc. of PDC'12, Vol. 2. ACM, New York, NY, USA, p. 17--20. DOI=http://dx.doi.org/10.1145/2348144.2348151 Google ScholarDigital Library
- Kensing, F. and Blomberg, J. 1998. Participatory Design: Issues and Concerns, in Journal of CSCW, Kluwer Academic Publishers 7: p. 167--185. Google ScholarDigital Library
- Kensing, F., Simonsen, J. and Bødker, K. 1996. MUST - a Method for Participatory Design, In Proc. of PDC'96, Cambridge, MA, p. 129--140.Google Scholar
- Madaus, G. F. and Stufflebeam, D. L. 2000. Program Evaluation: A Historical Overview. In Daniel L. Stufflebeam, George F. Madaus, Thomas Kellaghan (eds). Evaluation Models, Kluwer, pp3--18.Google Scholar
- Mambrey, P., Mark, G. and Pankoke-Babatz, U. 1996. In Proc. of PDC'96, Cambridge, MA, USA, p. 251--259.Google Scholar
- Mark, M. M. Greene, J. C. and Shaw, I. E. 2004. Introduction: The Evaluation of Policies, Programs, and Practices. In: The Sage Handbook of Evaluation, Melvin M. Mark, Jennifer C. Greene & Ian E. Shaw (eds), p. 1--30.Google Scholar
- Merkel, C. B., Xiao, L. Farooq, U. Ganoe, G. H., Lee, R., Carroll, J. M. and Rosson, M. B. 2004. Participatory design in community computing contexts: tales from the field. In Proc. of PDC 04, Vol. 1. ACM, New York, NY, USA, p. 1--10. DOI=http://dx.doi.org/10.1145/1011870.1011872 Google ScholarDigital Library
- Muller, M. and Druin, A. 2010. Participatory Design: The Third Space in Human-Computer Interaction, in The Human-Interaction Handbook. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Kapuire, G. K. Winschiers-Theophilus, H. and Blake, E. 2015. An insider perspective on community gains: A subjective account of Namibian rural communities' perception of a long-term participatory design project in International Journal of Human-Computer Studies, Volume 74, 124--143. Google ScholarDigital Library
- Patton, M. Q. 1997. Utilization-Focused Evaluation. Thousand Oaks, CA. Sage.Google Scholar
- Pollack, J. 2007. The changing paradigms of project management. International Journal of Project Management, 25, 266--274, DOI=10.1016/j.ijproman.2006.08.002Google ScholarCross Ref
- Power, M. 1997. The Audit Socity. Oxford, Oxford University PressGoogle Scholar
- Robertson, T. and Wagner, I. 2012. Ethics: engagement, representation and politics-in-action. in Routledge International Handbook of Participatory Design. Routledge, Taylor & Francis Ltd.Google Scholar
- Shadish, W. S., Cook, T. D. and Leviton, L. C. 1991. "Foundations of Program Evaluation", Newbury Park, Sage.Google Scholar
- Shapiro, D. 2005. Participatory design: the will to succeed. In Proc. of Critical computing (CC '05), ACM, New York, NY, USA, p. 29--38. DOI=http://dx.doi.org/10.1145/1094562.1094567 Google ScholarDigital Library
- Simonsen, J. and Hertzum, M. 2008. Participative design and the challenges of large-scale systems: extending the iterative PD approach. In Proc. of PDC '08. Indianapolis, IN, USA, p. 1--10. Google ScholarDigital Library
- Simonsen, J. and Hertzum, M. 2012. Real-use evaluation of effects: emergency departments aiming for 'Warm Hands'. In Proc. of PDC'12), Vol. 2. ACM, New York, NY, USA, p. 69--72. DOI=http://dx.doi.org/10.1145/2348144.2348166 Google ScholarDigital Library
- Simonsen, J. and Robertson, T. 2012. Routledge International Handbook of Participatory Design. Routledge, Taylor & Francis Ltd.Google Scholar
- Taxén, G. 2004. Introducing participatory design in museums. In Proc. of PDC 04, Vol. 1. ACM, New York, NY, USA, p. 204--213. DOI=http://dx.doi.org/10.1145/1011870.1011894 Google ScholarDigital Library
- Thoresen, K. 1990. Prototyping organizational change. In Proc. of PDC'90, p. 22--35.Google Scholar
- Timpka, T. and Sjöberg, C. 1994. Voices in Design: The Dynamics of Participatory Information System Design. In Proc. of PDC'94, Palo Alto, CA, USA p. 75--85.Google Scholar
- Whittle, J. 2014. How much participation is enough? A comparison of six participatory design projects in terms of outcomes. In Proc. of PDC '14, Vol. 1. ACM, New York, NY, USA, p. 121--130. DOI=http://dx.doi.org/10.1145/2661435.2661445 Google ScholarDigital Library
- Winchiers-Theophilus, H. 2006. The Challenges of Participatory Design in a Intercultural Context: Designing for Usability in Namibia, Proc. PDC '06, ACM, NY, p. 73--76.Google Scholar
Index Terms
- Evaluation in participatory design: a literature survey
Recommendations
What is a participatory design result?
PDC '16: Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1In this paper we discuss what the result of a Participatory Design (PD) process is and how it can be described and evaluated. We look at several PD projects and discuss if they have a participatory result and how we know that it is participatory. We ...
Applying participatory design theory to designing evaluation methods
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsSystem evaluators face several challenges in designing evaluation methods, including measurement and relevance, context, establishing common ground with users, and eliciting users' tacit knowledge. To address these challenges, we propose applying ...
The role of design fiction in participatory design processes
NordiCHI '18: Proceedings of the 10th Nordic Conference on Human-Computer InteractionParticipatory design is in essence very malleable as any design technique could lend itself to it, as long as users and stakeholders are involved. Design fictions however, have more often been used as either a vehicle for critical designs, or as a sheer ...
Comments