The practice of emergency medicine/brief research report
Accuracy and Reliability of Emergency Department Triage Using the Emergency Severity Index: An International Multicenter Assessment

https://doi.org/10.1016/j.annemergmed.2017.09.036Get rights and content

Study objective

We assess accuracy and variability of triage score assignment by emergency department (ED) nurses using the Emergency Severity Index (ESI) in 3 countries. In accordance with previous reports and clinical observation, we hypothesize low accuracy and high variability across all sites.

Methods

This cross-sectional multicenter study enrolled 87 ESI-trained nurses from EDs in Brazil, the United Arab Emirates, and the United States. Standardized triage scenarios published by the Agency for Healthcare Research and Quality (AHRQ) were used. Accuracy was defined by concordance with the AHRQ key and calculated as percentages. Accuracy comparisons were made with one-way ANOVA and paired t test. Interrater reliability was measured with Krippendorff’s α. Subanalyses based on nursing experience and triage scenario type were also performed.

Results

Mean accuracy pooled across all sites and scenarios was 59.2% (95% confidence interval [CI] 56.4% to 62.0%) and interrater reliability was modest (α=.730; 95% CI .692 to .767). There was no difference in overall accuracy between sites or according to nurse experience. Medium-acuity scenarios were scored with greater accuracy (76.4%; 95% CI 72.6% to 80.3%) than high- or low-acuity cases (44.1%, 95% CI 39.3% to 49.0% and 54%, 95% CI 49.9% to 58.2%), and adult scenarios were scored with greater accuracy than pediatric ones (66.2%, 95% CI 62.9% to 69.7% versus 46.9%, 95% CI 43.4% to 50.3%).

Conclusion

In this multinational study, concordance of nurse-assigned ESI score with reference standard was universally poor and variability was high. Although the ESI is the most popular ED triage tool in the United States and is increasingly used worldwide, our findings point to a need for more reliable ED triage tools.

Introduction

Accurate and reliable triage is needed to safely prioritize care and achieve optimal resource allocation in busy emergency departments (EDs). The 5-level Emergency Severity Index (ESI) is among the most popular ED triage algorithms, currently in use by a majority of EDs in the United States and increasingly adopted internationally.1, 2 The ESI facilitates rapid decisionmaking through 3 primary questions: (1) Does this patient require an immediate lifesaving intervention? (2) Is this a high-risk situation? and (3) How many resources are required to care for this patient?1

Editor’s Capsule Summary

What is already known on this topic

The Emergency Severity Index (ESI) is a simple, commonly used tool for emergency department (ED) triage categorization.

What question this study addressed

Does ESI use by nurses create accurate and consistent acuity designations in similar patients?

What this study adds to our knowledge

Using standard triage scenarios presented to 87 ESI-trained nurses in 3 countries, the mean accuracy was 59.2%, with only modest interrater reliability. Medium-acuity and adult patient scenarios had greater but still truncated accuracy.

How this is relevant to clinical practice

The ESI has limits when used to guide triage. Opportunity for a better initial ED triage tool exists.

The ESI is unique in its explicit consideration of resource use and use of a simple algorithm that is subjective and relies heavily on nursing intuition to facilitate rapid decisionmaking. In an era in which ED crowding is commonplace, with time and resources universally constrained, the utility of the ESI is obvious and its popularity is not surprising. However, it has not been well validated against time-sensitive or critical outcomes, and the majority of studies establishing ESI validity and reliability were performed soon after its implementation and in the United States, where the tool was developed.1 Several lines of evidence, including a wide range of interrater reliability,2 lack of true discrimination, with nearly half of patients nationwide triaged to the middle-tier ESI level 3,2, 3 and reports of suboptimal performance with low accuracy and high variability outside the United States,4, 5 suggest that the ESI may perform less well in true clinical context than expected.3, 6

In this internationally collaborative study, we sought to assess accuracy and variability of triage score assignment by ESI-trained nurses using standardized triage cases in 3 countries. We hypothesized suboptimal concordance with reference standard (ie, low accuracy) and high variability between individual raters across all sites. We also hypothesized higher accuracy among more experienced nurses and explored relationships between triage scenario type (ie, trauma versus nontrauma and adult versus pediatric) and accuracy and variability.

Section snippets

Study Design, Setting, and Selection of Participants

This cross-sectional multicenter study was conducted between December 2015 and December 2016 in the EDs of 3 community hospitals located in 3 countries: the United Arab Emirates, Brazil, and the United States. All study sites had used the ESI for ED triage for at least 2 years before study initiation, and participants were recruited from the pool of active triage nurses at each site. Responses were collected anonymously from volunteer participants, and there was no reward or penalty associated

Characteristics of Study Subjects

Site characteristics are shown in Table 1. A total of 87 nurses from the 3 study sites participated, with highest rates of participation achieved in the United Arab Emirates (87.5%) and Brazil (78.9%) and lowest in the United States (62.9%). Overall, participating nurses from the United Arab Emirates reported the most years of experience in nursing and ESI triage, whereas nurses from the United States reported the least (Table 1).

Mean accuracy pooled across all sites and scenarios was 59.2%

Limitations

This study has several important limitations. A thorough assessment of ESI validity should be performed in true clinical context and include analysis of the relationship between triage classification and actual ED resource utilization and clinical outcomes (eg, predictive accuracy). However, such an approach is resource intensive and beyond the scope of the current study, which aimed to measure consistency in application of the ESI algorithm by ED nurses trained and practicing in various

Discussion

In what is to our knowledge the first study designed to assess and compare accuracy and reliability of the ESI score assignment across multiple countries, we found that concordance of nurse-assigned ESI score with reference standard was universally poor and variability of score assignment was high within and across sites. Furthermore, supplemental within-groups accuracy analysis suggested consequential changes in care trajectory for 1 in 4 patients, with nearly 30% of high-acuity patients (ESI

References (11)

There are more references available in the full text version of this article.

Cited by (0)

Please see page 582 for the Editor’s Capsule Summary of this article.

Supervising editor: Donald M. Yealy, MD

Author contributions: BM, SSDR, KSB, and JSH conceived and designed the study. BM, SSDR, PSKS, KSB, and XA recruited participants. GK, DM, and KP provided statistical advice on the study design. DM and KP helped analyze the data. SL provided content expertise. BM and JSH drafted the article, and all authors contributed to its revision. JSH takes responsibility for the paper as a whole.

All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). This work was funded by a Synergy Award from Johns Hopkins University and grant R21HS023641 from the Agency for Healthcare Research and Quality. Dr. Levin is an engineer and the owner of a start-up company that has developed several machine-learning and electronic health record–based tools to improve patient safety and throughput throughout the hospital, including a tool that aims to improve reliability of ED triage.

The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of Johns Hopkins or the Agency for Healthcare Research and Quality.

View full text