Download PDF
Yearb Med Inform 2016; 25(01): 61-69
DOI: 10.15265/IY-2016-013
IMIA and Schattauer GmbH
Georg Thieme Verlag KG Stuttgart

Improving Evaluation to Address the Unintended Consequences of Health Information Technology

A Position Paper from the Working Group on Technology Assessment & Quality Development
F. Magrabi
1   Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
,
E. Ammenwerth
2   UMIT, University for Health Sciences, Medical Informatics and Technology, Hall in Tyrol, Austria
,
H. Hyppönen
3   National Institute for Health and Welfare, Information Department, Helsinki, Finland
,
N. de Keizer
4   Academic Medical Center, Department of Medical Informatics, Amsterdam, The Netherlands
,
P. Nykänen
5   University of Tampere, School of Information Sciences, eHealth Research, Finland
,
M. Rigby
6   Keele University, School of Social Science and Public Policy, Keele, United Kingdom
,
P. Scott
7   School of Computing, University of Portsmouth, Portsmouth, United Kingdom
,
J. Talmon
8   Hi-Way, Sint Odiliënberg, The Netherlands
,
A. Georgiou
1   Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
› Author Affiliations
Further Information

Publication History

10 November 2016

Publication Date:
06 March 2018 (online)

   Permissions and Reprints

Summary

Background and objectives: With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT.

Methods: Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation.

Conclusion: Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a pro-active and evidence-based approach to detecting and addressing the unintended effects of health IT.

Keywords

Medical informatics - evaluation studies - evidence-based medicine - information systems - patient safety
 

  • References

  • 1 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-23.
    CrossrefPubMedGoogle Scholar
  • 2 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-12.
    PubMedGoogle Scholar
  • 3 Weiner JP, Kfuri T, Chan K, Fowles JB. “e-Iatro-genesis”: the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007; 14 (03) 387-8.
    CrossrefPubMedGoogle Scholar
  • 4 Rahadhan P, Poon SK, Land L. Understanding unintended consequences for EMR: a literature review. Stud Health Technol Inform 2012; 178: 192-8.
    PubMedGoogle Scholar
  • 5 Kuperman GJ, McGowan JJ. Potential unintended consequences of health information exchange. J Gen Intern Med 2013; 28 (12) 1663-6.
    CrossrefPubMedGoogle Scholar
  • 6 Campbell EM, Sittig DF, Guappone KP, Dykstra RH, Ash JS. Overdependence on technology: an unintended adverse consequence of computerized provider order entry. AMIA Annu Symp Proc 2007; 94-8.
    PubMedGoogle Scholar
  • 7 Bloomrosen M, Starren J, Lorenzi NM, Ash JS, Patel VL, Shortliffe EH. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. J Am Med Inform Assoc 2011; 18 (01) 82-90.
    CrossrefPubMedGoogle Scholar
  • 8 Coiera E, Aarts J, Kulikowski C. The dangerous decade. J Am Med Inform Assoc 2012; 19 (01) 2-5.
    CrossrefPubMedGoogle Scholar
  • 9 ECRI Institute’s 2016 Top 10 Health Technology Hazards. ECRI Institute; USA:
    Google Scholar
  • 10 Magrabi F, Baker M, Sinha I, Ong MS, Harrison S, Kidd MR. et al. Clinical safety of England’s national programme for IT: A retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84 (03) 198-206.
    CrossrefPubMedGoogle Scholar
  • 11 Magrabi F, Ong MS, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (01) 45-53.
    CrossrefPubMedGoogle Scholar
  • 12 Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press: Institute of Medicine; 2012
    Google Scholar
  • 13 Kuziemsky CE. Review of Social and Organizational Issues in Health Information Technology. Healthc Inform Res 2015; 21 (03) 152-60.
    CrossrefPubMedGoogle Scholar
  • 14 Ammenwerth E, Shaw NT. Bad health informatics can kill--is evaluation the answer?. Methods Inf Med 2005; 44 (01) 1-3.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 15 Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC. et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-12.
    CrossrefPubMedGoogle Scholar
  • 16 HI B. Bad Health Informatics can Kill. [Available from: http://iig.umit.at/efmi.
    PubMedGoogle Scholar
  • 17 Rigby M, Ammenwerth E, Beuscart-Zephir MC, Brender J, Hypponen H, Melia S. et al. Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Medical Inform 2013; 8: 34-46.
    PubMedGoogle Scholar
  • 18 Ammenwerth E, Rigby M. editors. Evidence-Based Health Informatics - Promoting Safety and Efficiency through Scientific Methods and Ethical Policy. Amsterdam: IOS Press; 2016
    Google Scholar
  • 19 Wall R. Computer Rx: more harm than good?. Journal Med Syst 1991; 15 5-6 321-34.
    PubMedGoogle Scholar
  • 20 Nykänen P, Kaipio J. Quality of health IT evaluations. In: Ammerwerth E, Rigby M. editors. Evidence-based health informatics. Stud Health Technol Inform. 2016. 222: 291-303.
    Google Scholar
  • 21 Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform 2008; 77 (06) 377-85.
    CrossrefPubMedGoogle Scholar
  • 22 Basden A. Philosophical frameworks for understanding information systems. Hershey, PA, USA: IGI Global; 2008
    Google Scholar
  • 23 House ER. Origins of the Ideas in Evaluating with Validity. New Directions for Evaluation. 2014; 2014 (142) 9-15.
    PubMedGoogle Scholar
  • 24 Friedman CF, Wyatt J. Evaluation methods in medical informatics, 2 Ed. New York: Springer Verlag; 2006
    Google Scholar
  • 25 Delone WH, McLean ER. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manage Inf Syst 2003; 19 (04) 9-30.
    CrossrefPubMedGoogle Scholar
  • 26 Venkatesh V, Morris M, Davis G, Davis F. User acceptance of information technology: toward a unified view. MIS Quarterly 2003; 27 (03) 425-78.
    CrossrefPubMedGoogle Scholar
  • 27 Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak 2006; 6: 3.
    CrossrefPubMedGoogle Scholar
  • 28 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (01) 21-7.
    CrossrefPubMedGoogle Scholar
  • 29 Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc 2007; 14 (05) 542-9.
    CrossrefPubMedGoogle Scholar
  • 30 Cusack C, Byrne C, Hook J, McGowan J, Poon E, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update. Rockville, MD: Agency for Healthcare Research and Quality; 2009. [updated June. Available from: https://healthit.ahrq.gov/sites/default/files/docs/page/health-information-technology-evaluation-tool-kit-2009-update.pdf.
    Google Scholar
  • 31 Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation 2012; 18 (02) 176-91.
    CrossrefPubMedGoogle Scholar
  • 32 De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C. et al. Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials 2014; 15: 267.
    CrossrefPubMedGoogle Scholar
  • 33 Taplin DH, Clark H, Collins E, Colby DC. Theory of Change technical papers New York: ActKnowl-edge. 2013 [updated 31 July 2015. Available from: http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/ToC-Tech-Papers.pdf.
    PubMedGoogle Scholar
  • 34 Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC. et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011; 80 (12) 815-27.
    CrossrefPubMedGoogle Scholar
  • 35 Brender J. Handbook of evaluation methods for health informatics. USA: Elsevier Academic Press; 2006
    Google Scholar
  • 36 Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf Med 2004; 43 (03) 215-31.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 37 Hyppönen H, Kangas M, Reponen J, Nøhr C, Villumsen S, Koch S. et al. Nordic eHealth Benchmarking: Status 2014: Nordisk Ministerråd. 2015
    PubMedGoogle Scholar
  • 38 Lääveri T, Vainiomäki S, Kaipio J, Reponen J, Vänskä J, Lehtovirta M. et al. Yksityissektorin potilastietojärjestelmät arvioitu 2014 [Evaluation of private sector patient information systems 2014]. Finnish Medical Journal 2015; 23: 1660-7.
    PubMedGoogle Scholar
  • 39 Hyppönen H, Hämäläinen P, Reponen J. editors. E-health and e-welfare of Finland. Check point 2015. Helsinki, Finland: National Institute for Health and Welfare; 2015
    Google Scholar
  • 40 Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009; 78 (01) 1-9.
    CrossrefPubMedGoogle Scholar
  • 41 Brender J, Talmon J, de Keizer N, Nykanen P, Rigby M, Ammenwerth E. STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration. Appl Clin Inform 2013; 4 (03) 331-58.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 42 de Keizer NF, Talmon J, Ammenwerth E, Brender J, Rigby M, Nykanen P. Systematic prioritization of the STARE-HI reporting items. An application to short conference papers on health informatics evaluation. Methods Inf Med 2012; 51 (02) 104-11.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 43 EQUATOR.. The Equator Network: Enhancing the Quality and Transparency of Health Research. 2014 [Available from: http://www.equator-network.org.
    PubMedGoogle Scholar
  • 44 Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002. Methods Inf Med 2005; 44 (01) 44-56.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 45 Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W. et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med 2010; 49 (02) 105-20.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 46 Eval EW. Curricula of health IT evaluation courses. 2014 [Available from:. https://iig.umit.at/efmi/curricula.htm.
    PubMedGoogle Scholar
  • 47 Balogh E, Miller B, Ball J. Improving Diagnosis in Health Care. Institute of Medicine. National Academies Press; 2015
    Google Scholar
  • 48 Poon EG, Cusack CM, McGowan JJ. Evaluating healthcare information technology outside of academia: observations from the national resource center for healthcare information technology at the Agency for Healthcare Research and Quality. J Am Med Inform Assoc 2009; 16 (05) 631-6.
    CrossrefPubMedGoogle Scholar
  • 49 Rigby M, Georgiou A, Hypponen H, Ammenwerth E, de Keizer N, Magrabi F. et al. Patient Portals as a Means of Information and Communication Technology Support to Patient- Centric Care Coordination - the Missing Evidence and the Challenges of Evaluation. A joint contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Med Inform 2015; 10 (01) 148-59.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 50 Ijzerman MJ, Steuten LM. Early assessment of medical technologies to inform product development and market access: a review of methods and applications. Appl Health Econ Health Policy 2011; 9 (05) 331-47.
    CrossrefPubMedGoogle Scholar