Simulation and educationValidation for a scoring system of the ALS cardiac arrest simulation test (CASTest)☆
Introduction
A variety of healthcare professionals need to demonstrate competency in all aspects of advanced life support (ALS).1, 2 It is vital that certification of such skills is based on robust testing, giving ALS providers credibility and enabling them to promote patient safety. As with any education test, evidence of validity is key if the results are to be interpreted meaningfully.3 Assessments are not valid or invalid in themselves but rather a certain interpretation is more or less valid for a certain population at a certain point in time.4 Evidence should be collected from multiple sources and analysed to create arguments for and against a specific interpretation of test results.
The ALS course teaches both theoretical and practical aspects of resuscitation. Participants complete a pre-course multiple choice question (MCQ) test based on pre-course preparation. During the course participants are assessed on practical skills involving airway management and the initial approach to critically ill patients. Summative assessment at the end of the course is a combination of an MCQ paper (pass mark 75%) and a cardiac arrest simulation test (CASTest).1
The CASTest uses a simulated cardiac arrest to test the application of resuscitation knowledge, and skills and is focussed on team leadership and decision making.5 The score sheet contains performance criteria that participants are expected to demonstrate in order to pass the test. Performance criteria are classified as bold (i.e. essential) and non-bold (desirable). Based on this structured assessment of performance the assessment outcome is reported as a single binary pass–fail mark. We have developed a four point scoring system which can be applied to each of the 24 performance criteria in order to characterise the quality of candidate performance in more detail. The scoring system does not give differential score for bold and non-bold treatment points.
The aim of this study is to determine the measurement properties of the new tool by considering the tool's internal structure and relationship with other variables (i.e. theoretical (post-course MCQ paper) and practical skill assessments) and thus report its utility to better characterise performance in CASTest than a simple pass/fail result.
Section snippets
Participants
The study was approved by South Birmingham Research Ethics Committee. Participants enrolled in an evaluation of pre-course computer simulation material from ALS courses at 11 UK centres were eligible for inclusion in the study. Participants provided written informed consent. The study was conducted between March and December 2007.
Assessment criteria
Participants underwent assessment of the following aspects: pre-/post-course MCQ paper, skills assessments (airway, initial assessment and resuscitation), and a
Results
A total of 537 participants were assessed: 346 doctors, 97 nurses, 7 operating department practitioners, 8 others and 79 unknown. There were more than double the number of junior staff compared to senior staff (267 vs 114), whilst only 27 were students and 129 unknown. Data for pre-course MCQ were available from 429 (79.9%) of participants.
Discussion
This study presents evidence supporting the validity of a scoring system which can be used to measure overall performance during advanced life support cardiac arrest simulation testing. This system will allow greater precision in classifying performance than the existing binary pass or fail outcome measure. Using a large cohort of multi-professional health care providers the participants awarded a pass mark by the standard assessment tool obtained significantly higher performance scores than
Conclusions
Evidence supporting the construct validity of the CASTest scoring system is presented. This simple scoring system better characterises performance in the ALS course CASTest than the current binary pass–fail outcome.
Conflict of interest statement
Gavin Perkins, Andy Lockey, Ian Bullock and Robin Davies are non-paid members of the Resuscitation Council UK ALS committee responsible for the development of the ALS course and its assessments.
Robin Davies is employed by the Resuscitation Council (UK).
Acknowledgments
We would like to thank the Resuscitation Research Group for assistance with the project: Nicole Gomez-Davis, Marie Fletcher, Teresa Melody, Fang Gao and Dawn Hill.
GDP holds a DH (NIHR) Clinician Scientist and is supported by the Intensive Care Foundation. The project was funded by the Resuscitation Council (UK) and Laerdal Foundation for Acute Medicine. The study design, analysis and decision to publish were the responsibility of the authors.
Collaborators: Julie Lawrence, Good Hope Hospital,
References (14)
Simulation in resuscitation training
Resuscitation
(2007)- et al.
Evaluation of staff's retention of ACLS and BLS skills
Resuscitation
(2008) - et al.
Assessment of advanced life support competence when combining different test methods—reliability and validity
Resuscitation
(2007) - et al.
Advanced life support cardiac arrest scenario test evaluation
Resuscitation
(2007) - et al.
Variability in the assessment of advanced life support skills
Resuscitation
(2001) - et al.
The advanced life support provider course
BMJ
(2002) - et al.
European Resuscitation Council guidelines for resuscitation 2005, Section 9. Principles of training in resuscitation
Resuscitation
(2005)
Cited by (34)
Feedback in advanced life support: A quality improvement initiative
2020, ResuscitationAdvanced life support provider course in Italy: A 5-year nationwide study to identify the determinants of course success
2015, ResuscitationCitation Excerpt :Accordingly, in our study, being an MD and having a high pre-course MCQ score were independent predictors of course success. It is reasonable to expect that clinical experience might improve knowledge and the levels of resuscitation competence.19,22,23 Indeed, nurses and doctors working in high-risk areas for cardiac arrest or encountering more cardiac arrest events have demonstrated greater BLS and ALS theoretical knowledge compared to their colleagues working in lesser exposed areas.7,21,24
- ☆
A Spanish translated version of the abstract of this article appears as Appendix in the final online version at doi:10.1016/j.resuscitation.2009.04.043.