The practice of emergency medicine/original research
Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index

https://doi.org/10.1016/j.annemergmed.2017.08.005Get rights and content

Study objective

Standards for emergency department (ED) triage in the United States rely heavily on subjective assessment and are limited in their ability to risk-stratify patients. This study seeks to evaluate an electronic triage system (e-triage) based on machine learning that predicts likelihood of acute outcomes enabling improved patient differentiation.

Methods

A multisite, retrospective, cross-sectional study of 172,726 ED visits from urban and community EDs was conducted. E-triage is composed of a random forest model applied to triage data (vital signs, chief complaint, and active medical history) that predicts the need for critical care, an emergency procedure, and inpatient hospitalization in parallel and translates risk to triage level designations. Predicted outcomes and secondary outcomes of elevated troponin and lactate levels were evaluated and compared with the Emergency Severity Index (ESI).

Results

E-triage predictions had an area under the curve ranging from 0.73 to 0.92 and demonstrated equivalent or improved identification of clinical patient outcomes compared with ESI at both EDs. E-triage provided rationale for risk-based differentiation of the more than 65% of ED visits triaged to ESI level 3. Matching the ESI patient distribution for comparisons, e-triage identified more than 10% (14,326 patients) of ESI level 3 patients requiring up triage who had substantially increased risk of critical care or emergency procedure (1.7% ESI level 3 versus 6.2% up triaged) and hospitalization (18.9% versus 45.4%) across EDs.

Conclusion

E-triage more accurately classifies ESI level 3 patients and highlights opportunities to use predictive analytics to support triage decisionmaking. Further prospective validation is needed.

Introduction

Increases in emergency department (ED) visits to more than 130 million annually in the United States have led to unprecedented levels of crowding and delays in care.1, 2 Evidence linking delays to increased morbidity, mortality, and poor process measures has been established across many clinical conditions.3, 4, 5, 6, 7, 8, 9, 10 Critically ill patients are most vulnerable to worse health outcomes because of delays.11, 12, 13

Editor’s Capsule Summary

What is already known on this topic

Emergency Severity Index (ESI) 5-level triage requires its own resources and tends to batch patients in an undifferentiated middle category. Automated triage systems represent a new approach.

What question this study addressed

How would an electronic triage system that generated triage categories through a machine learning algorithm compare with ESI 5-level triage?

What this study adds to our knowledge

The computer-supported approach better predicted need for hospitalization and critical care.

How this is relevant to clinical practice

Refinement of triage can better assist in emergency department flow and patient safety. The process developed here allowed greater differentiation of heterogenous ESI level 3 patients.

Triage often presents the first opportunity to identify critically ill patients and tends to set the trajectory for further ED care. It drives ED patient care location, queue position, and timing, and it influences provider decisionmaking (ie, resource use) up to and including final disposition.14, 15, 16, 17 Thus, crowded EDs must maintain accurate triage systems to quickly identify and prioritize patients with critical conditions from the volumes of those with less urgent needs. Although simple in concept, the practice of triage is challenging because of limited information, time pressure, diverse medical conditions, and heavy reliance on intuition. As a result, the projected clinical course at presentation (ie, triage) is not obvious for the majority of ED patients. Almost half of adult ED visits nationally and more than 65% of ED visits at our urban and community sites were triaged to Emergency Severity Index (ESI) level 3, the ambiguous midpoint of a 5-level triage algorithm now standard in the United States.1, 18, 19, 20

Despite the ESI’s widespread adoption, it relies heavily on provider judgment, subject to high variation.21 It also poorly differentiates a large diversely ill patient group (ESI level 3), counter to the objective of triage.18, 20 Inability to differentiate poses safety risks to patients critically ill and undertriaged and may influence the precision and efficiency of ED resource allocation (ie, overutilization) because low-acuity patients are overtriaged.22, 23, 24, 25 The patient safety challenges associated with triage in crowded EDs, limitations of ESI in practice, and need for accurate risk assessment motivated us to develop an electronic health record–driven electronic triage system that leverages machine-learning capability.

Machine learning is a set of computational methods that learn patterns in data without being explicitly programmed.26, 27 These methods offer advantages for predictive clinical applications because they can be designed to yield stable predictions,28, 29 are able to perform variable selection as part of the model building process,29 are flexible in handling predictor data favorable for electronic health record applications,29, 30 and are adept at identifying interactions in patient information, enabling them to define patient subgroups with respect to predicted outcomes.30 The last characteristic makes them useful in analyzing patients across a wide spectrum of medical conditions and illness severity germane to triage and the practice of emergency medicine in general.

The objective of this study was to use machine-learning methods to develop an electronic triage support system (e-triage) that predicts clinically important patient outcomes and facilitates differentiation of current midacuity (ESI level 3) patients. Our hypothesis was that e-triage would support improved patient differentiation compared with ESI and be adaptable to EDs’ individual populations and patient distribution objectives.

Section snippets

Setting and Selection of Participants

E-triage was developed from a retrospective cohort of 172,726 adult (≥18 years) visits from an urban academic ED (60,712) and a community ED (112,014) between August 2014 and October 2015, and June 2013 and October 2015, respectively. Each ED’s annual volume varies between 60,000 and 70,000 visits per year. From the adult ED visit cohort receiving a final disposition, patients were excluded who presented with psychiatric conditions, had unknown complaint data, or had missing vital signs (

Results

ED patient visit outcomes and predictor characteristics are displayed in Table 1. The rates of critical care (2.0% urban ED; 1.6% community ED), emergency procedures (1.4% urban ED; 1.7% community ED), and hospitalization (26% urban ED; 22.3% community ED) were similar across EDs. However, rates of secondary clinical markers for elevated troponin and lactate levels were higher for the urban ED compared with the community ED. Demographics (age and sex), the proportion of patients with vital sign

Limitations

There are several limitations to this study and the e-triage tool itself. Foremost, this evaluation was based on retrospective data that solely provide evidence for the opportunity to improve triage by using e-triage. This study in no way validates the tool’s performance prospectively. In addition, retrospective data are always subject to potential error in data entry. This was mitigated through data verification and processing that included chart review by clinician team members at the urban

Discussion

In this study, we developed a machine-learning-based triage tool (e-triage) and evaluated its performance in multiple EDs, using ESI as a comparator. E-triage demonstrated equivalent or improved identification of patients with critical outcomes (mortality, ICU admission, and emergency procedure), hospital admission, and secondary measures of elevated troponin and lactate levels.

Outcomes predicted by e-triage are simple indicators for high-severity medical need that span a broad range of

References (54)

  • J.A. Osheroff et al.

    A roadmap for national action on clinical decision support

    J Am Med Inform Assoc

    (2007)
  • Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care Survey: 2013 emergency department...
  • American Hospital Association. Chart 3.7: emergency department visits and emergency departments in community hospitals...

  • Hospital-Based Emergency Care: At the Breaking Point

    (2006)
  • S.L. Bernstein et al.

    The effect of emergency department crowding on clinically oriented outcomes

    Acad Emerg Med

    (2009)
  • E.B. Kulstad et al.

    Overcrowding is associated with delays in percutaneous coronary intervention for acute myocardial infarction

    Int J Emerg Med

    (2009)
  • J.M. Pines et al.

    The association between emergency department crowding and hospital performance on antibiotic timing for pneumonia and percutaneous intervention for myocardial infarction

    Acad Emerg Med

    (2006)
  • C. Fee et al.

    Effect of emergency department crowding on time to antibiotics in patients admitted with community-acquired pneumonia

    Ann Emerg Med

    (2007)
  • D.B. Chalfin et al.

    Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit

    Crit Care Med

    (2007)
  • G. Intas et al.

    The impact of ED boarding time, severity of illness, and discharge destination on outcomes of critically ill ED patients

    Adv Emerg Nurs J

    (2012)
  • France D, Levin S, Ding R, et al. Factors influencing time-dependent quality indicators for suspected acute coronary...
  • Dugas A, Kirsch T, Toerper M, et al. An electronic emergency triage system improves patient distribution. J Emerg Med....
  • M. McHugh et al.

    More patients are triaged using the Emergency Severity Index than any other triage acuity system in the United States

    Acad Emerg Med

    (2012)
  • R. Arya et al.

    Decreasing length of stay in the emergency department with a split Emergency Severity Index 3 patient flow model

    Acad Emerg Med

    (2013)
  • M. Christ et al.

    Modern triage in the emergency department

    Dtsch Arztebl Int

    (2010)
  • M. Hitchcock et al.

    Triage: an investigation of the process and potential vulnerabilities

    J Adv Nurs

    (2014)
  • N. Seiger et al.

    Undertriage in the Manchester triage system: an assessment of severity and options for improvement

    Arch Dis Child

    (2011)

    • Cited by (0)

    Please see page 566 for the Editor’s Capsule Summary of this article.

    Supervising editors: Stephen Schenkel, MD, MPP; Robert L. Wears, MD, PhD†

    Author contributions: SL, MT, AD, TK, and GK were responsible for the original development of the triage tool evaluated in this study. SL, MT, JH, and SB contributed to the study design. SL, MT, and SB analyzed the data and provided statistical advice. EH, JH, HG, and BL contributed to understanding of implications of the tool in practice, including insights into how the ESI is used at study sites. SL drafted the article and all authors contributed to its revision. SL takes responsibility for the paper as a whole.

    All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

    Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). This study was supported by Agency for Healthcare Research and Quality award R21HS023641 and National Science Foundation (NSF) Engineering Directorate award SBIR 1621899. A patent application for electronic triage has been filed by Johns Hopkins University (JHU). Drs. Levin, Barnes, and Dugas and Mssrs. Toerper and Hamrock have been supported by an NSF Small Business Innovation Research award to commercialize electronic triage toward improving ED crowding. This award was granted to a JHU start-up company cofounded by Dr. Levin and Mr. Hamrock, with JHU as an equity partner.

    †

    Dr. Wears died shortly before this article was accepted.

    View full text
    © 2017 by the American College of Emergency Physicians.