ResearchTriage Tool Inter-rater Reliability: A Comparison of Live Versus Paper Case Scenarios
Section snippets
Objective
This study was designed to assess the correlation between triage assessments conducted on patients in a live ED setting and those assigned to equivalent paper case scenarios. In addition, we aimed to compare the inter-rater reliability of the tool across both presentation media.
Ethics
This study protocol was submitted to the local Ethics Review Board and found to be exempt from formal ethics review because it was considered a quality assurance study that did not interfere with or alter standard
Results
Triage data were collected (and turned into paper-based cases) for a total of 271 patients. The IRR for the average rating assigned by 3 triage nurses when viewing the cases live was 0.9 (95% CI = 0.89-0.91). When the same cases were reviewed in paper format, the IRR for the average rating assigned by 3 triage nurses was 0.76 (95% CI = 0.74-0.78). The correlation between the research nurses' live triage assessments and the triage assessments they assigned to the paper cases was r = 0.73 (95% CI
Discussion
This study was designed to ensure that selected patients were representative of actual ED case mixes and employed “experienced” triage nurses representative of those performing triage assessments in other hospitals.
The advantages of triage researchers using paper case scenarios are in the savings of cost and time; rather than have a few triage nurses assess a few patients over a long period, multiple cases can be assessed by a large group of observers in a few hours. Although the results show a
Limitations
A limitation of this study is that the research triage nurses were not permitted to directly question patients, because this questioning would interfere with standard patient care. This restriction may have affected their initial triage assessment. However, the level of agreement found between the research nurses' triage assessments and actual triage nurse's assessments was high at 0.71. Also, the scoring of paper case scenarios similarly precluded acquiring additional information other than
Conclusions
Use of paper case scenarios to determine IRR of triage scales is an efficient method that estimates that of live cases. If the results are found to be within an acceptable performance range, further testing of IRR using live cases may be unnecessary.
Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
References (9)
- et al.
Paper case scenarios in the assessment of triage tools
Ann Emerg Med
(2005) - et al.
Reliability of the Canadian Emergency Department Triage and Acuity Scale: inter-observer agreement
Ann Emerg Med
(1999) Emergency department triage: why we need a research agenda
Ann Emerg Med
(2004)- et al.
Health Measurement Scales: a practical guide to their development and use (3rd ed.)
(2003)
Cited by (55)
A descriptive study of registered nurses’ application of the triage scale RETTS©; a Swedish reliability study
2018, International Emergency NursingValidity of ED: Addressing heterogeneous definitions of over-triage and under-triage
2017, American Journal of Emergency MedicineAcuity Assessment in Obstetrical Triage
2016, Journal of Obstetrics and Gynaecology CanadaPredicting Pediatric Emergency Severity Index Level Based on Emergency Department Pre-Arrival Information
2018, Journal of Pediatric NursingCitation Excerpt :The levels range from ESI Level 1, which is assigned to patients who require immediate life-saving interventions, to ESI Level 5, which is assigned to low acuity patients who are predicted to need no interventions (Gilboy et al., 2011). In previous studies, ESI Version 4 has been found to be a valid, reliable tool with good inter-rater reliability (Dateo, 2013; Green et al., 2012; Worster et al., 2007). The reliability of the tool in pediatrics was found to be moderate when studied in emergency departments that see both adult and pediatric populations (Travers, Waller, Katznelson, & Agans, 2009).
First responder accuracy using SALT after brief initial training
2015, Prehospital and Disaster Medicine
Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
Arlene Sardo is Nurse Practitioner, Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
Kevin Eva is Epidemiologist, Department of Clinical Epidemiology and Biostatics, McMaster University, Hamilton, Ontario, Canada.
Christopher M.B. Fernandes is Professor of Emergency Medicine, University of Western Ontario, London, Ontario.
Suneel Upadhye is Undergraduate Coordinator, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
This study was presented at the 2005 Society of Academic Emergency Medicine Annual Meeting, May 22-25, 2005, New York, NY; the Canadian Association of Emergency Physicians Annual Scientific Meeting, May 30, 2005, Edmonton, Alberta; and the ENA Annual Conference, Nashville, Tenn, September 14-17, 2005.
Funded by a research grant to Dr. Andrew Worster from the Hamilton Emergency Services Network, Hamilton Ontario, Canada.
Authors' note
This study has been published in abstract form as follows:
Sardo A, Worster A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Acad Emerg Med 2005;12:137.
Worster A, Sardo A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Can J Emerg Med 2005;7:209.
Worster A, Sardo A, Fernandes C, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. J Emerg Nurs 2005;31:426.