Integrating Quantitative and Qualitative Methods in Patient Care Information System Evaluation

 

 Buy Article Permissions and Reprints

Summary

Objective: The aim of this paper is twofold. First, we describe two important dimensions of patient care information systems (PCIS) evaluation: the domain of evaluation and the different phases of the PCIS implementation. Second, we claim that, though Randomized Controlled Trials (RCTs) are often still seen as the standard approach, this type of design hardly generates relevant information for the organizational decision maker.

Method: Interpretive study of evaluation literature. Results and Conclusions: The field of evaluation is scattered and the types of questions that can be asked and methods that can be used seem infinite and badly demarcated. Different stakeholders, moreover, often have different priorities in evaluating ICT. The most important reason for the lack of relevance of RCTs is that they are ill suited for investigating why and how a PCIS is being used, or not, and what the (often unplanned) effects and consequences are. Subsequently, our aim is to contribute to the discussion about the viability of qualitative versus quantitative methods in PCIS evaluation, by arguing for a specific integration of quantitative and qualitative research methods. The joint utilization of these methods, we claim, yields the richest results.

• References

• 1 Talmon J, Enning J, Castañeda G, Eurlings F, Hoyer D, Nykänen P. et al. The VATAM guidelines. Int J Med Inf 1999; 56: 107-15.
  CrossrefPubMedGoogle Scholar
• 2 Southon G. IT, Change and evaluation: an overview of the role of evaluation in health services. Int J Med Inf 1999; 56: 125-33.
  CrossrefPubMedGoogle Scholar
• 3 Heathfield HA, Buchan IA. Current evaluations of information technology in health care are often inadequate. BMJ 1996; 313: 1008.
  PubMedGoogle Scholar
• 4 Greatbatch D, Murphy E, Dingwall R. Evaluating medical information systems: ethnomethodological and interactionist approaches. Health Services Management Research 2001; 14 (Suppl. 03) 181-91.
  CrossrefPubMedGoogle Scholar
• 5 Savage J. Ethnography and health care. BMJ 2000; 321: 1400-2.
  CrossrefPubMedGoogle Scholar
• 6 Heathfield H, Hudson P, Kay S, Mackay L, Marley T, Nicholson L. et al. Issues in multi-disciplinary assessment of healthcare information systems. Information Technology & People 1999; 12: 253-75.
  CrossrefPubMedGoogle Scholar
• 7 Jones MR. Mission impossible? Pluralism and multi-paradigm IS research. Information Systems Review 2000; 1: 217-32.
  PubMedGoogle Scholar
• 8 Mingers J. Combining IS research methods: towards a pluralist methodology. Information Systems Research 2001; 12: 240-59.
  CrossrefPubMedGoogle Scholar
• 9 Øvretveit J. Evaluating health interventions. Buck-ingham: Open University Press; 1998
  Google Scholar
• 10 Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag; 1997
  Google Scholar
• 11 Kaplan B, Maxwell J. Qualitative research methods for evaluating computer information systems. In Anderson J, Aydin C, Jay S. editors. Evaluating Health Care Information Systems. California: Sage; 1994. pp. 45-69.
  Google Scholar
• 12 Van der Loo R. Overview of published assessment and evaluation studies. In: Van Gennip E, Talmon J. editors. Assessment and evaluation of information technologies. Amsterdam: IOS Press; 1995. pp. 261-83.
  Google Scholar
• 13 Kaplan B. Evaluating informatics applications, clinical decision support systems literature review. Int J Med Inf 2001; 64: 15-37.
  CrossrefPubMedGoogle Scholar
• 14 Kaplan B, Duchon D. Combining qualitative and quantitative methods in information systems research, a case study. MIS Quarterly 1988; 571-86.
  PubMedGoogle Scholar
• 15 Friedman CP, Elstein AE, Wolf FM, Murphy GC, Franz TM, Heckerling PS. et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation, a multi site study of two systems. JAMA 1999; 282: 19
  CrossrefPubMedGoogle Scholar
• 16 Kuperman GJ, Teich JM, Tanasijevic MJ, Luf NM, Rittenberg E, Jha A. et al. Improving response to critical laboratory results with automation, results of a randomized controlled trial. J Am Med Inform Ass 1999; 6: 512-22.
  PubMedGoogle Scholar
• 17 Berwick DM. Developing and testing changes in delivery of care. Ann Intern Med 1998; 128: 651-56.
  CrossrefPubMedGoogle Scholar
• 18 Anderson J, Aydin C, Jay S. editors. Evaluating health care information systems. Thousand Oaks (CA): Sage; 1994
  Google Scholar
• 19 Kaplan B. Addressing organizational issues into the evaluation of medical systems. J Am Med Inform Ass 1996; 4: 94-101.
  PubMedGoogle Scholar
• 20 Pope C. Qualitative research, reaching the parts other methods cannot reach, an introduction to qualitative methods in health and health services research. BMJ 1995; 311: 42-5.
  CrossrefPubMedGoogle Scholar
• 21 Jørgensen T. Measuring effects. In: Van Gennip E, Talmon J. editors. Assessment and evaluation of information technologies in medicine. Amsterdam: IOS Press; 1994. pp. 99-111.
  Google Scholar
• 22 Serafeimidis V, Smithson S. Information systems evaluation in practice: a case study of organizational change. J Inf Techn 2000; 15: 93-105.
  PubMedGoogle Scholar
• 23 Heathfield HA, Peel V, Hudson P, Kay S, Mackay L, Marley T. et al. Evaluating large scale health information systems, from practice towards theory. Proc AMIA Annu Fall Symp 1997; 116-20.
  PubMedGoogle Scholar
• 24 Farbey B, Land F, Targett D. The moving staircase, problems of appraisal and evaluation in a turbulent environment. Information Technology & People 1999; 12: 238-52.
  CrossrefPubMedGoogle Scholar
• 25 Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J Med Inf 1999; 55: 87-101.
  CrossrefPubMedGoogle Scholar
• 26 Hebert M. Professional and organizational impact of using patient care information systems. Medinfo 1998; 9: 849-53.
  PubMedGoogle Scholar