Elsevier

Resuscitation

Volume 80, Issue 9, September 2009, Pages 1034-1038
Resuscitation

Simulation and education
Validation for a scoring system of the ALS cardiac arrest simulation test (CASTest)

https://doi.org/10.1016/j.resuscitation.2009.04.043Get rights and content

Abstract

Aim

The cardiac arrest simulation test (CASTest) assesses resuscitation knowledge and skills during a simulated cardiac arrest. The aim of this study is to validate an alternative scoring system for measuring individual candidate performance during research involving the CASTest.

Methods

The performance of 537 participants was measured using the new scoring system. Evidence of internal structure was sought by comparing the score with global rating of performance and pass/fail decision; identification of participants with instructor potential, skill tests and MCQ scores. Relationships between CASTest score, profession and seniority were also examined.

Results

Global assessment of performance identified 413 passes (76.9%) and 124 fails (23.1%). CASTest score was significantly higher in those that passed than in those that failed (median 77 vs 62.5, P < 0.0001). There were no differences between professions. Senior staff performed slightly better than junior staff (median 74 and 72 respectively, P = 0.01). Excellent participants (identified as having instructor potential) scored significantly higher than the other participants (median 94 and 72 respectively, P < 0.0001). A strong correlation was demonstrated between domains in the CASTest (rho 0.72–0.82, P < 0.01). Other assessment outcomes for the ALS course correlated poorly with CASTest scores (rho 0.27–0.37, P < 0.01).

Conclusion

This new simple scoring system can be used to better characterise performance on the ALS course CASTest than the current binary pass–fail outcome.

Introduction

A variety of healthcare professionals need to demonstrate competency in all aspects of advanced life support (ALS).1, 2 It is vital that certification of such skills is based on robust testing, giving ALS providers credibility and enabling them to promote patient safety. As with any education test, evidence of validity is key if the results are to be interpreted meaningfully.3 Assessments are not valid or invalid in themselves but rather a certain interpretation is more or less valid for a certain population at a certain point in time.4 Evidence should be collected from multiple sources and analysed to create arguments for and against a specific interpretation of test results.

The ALS course teaches both theoretical and practical aspects of resuscitation. Participants complete a pre-course multiple choice question (MCQ) test based on pre-course preparation. During the course participants are assessed on practical skills involving airway management and the initial approach to critically ill patients. Summative assessment at the end of the course is a combination of an MCQ paper (pass mark 75%) and a cardiac arrest simulation test (CASTest).1

The CASTest uses a simulated cardiac arrest to test the application of resuscitation knowledge, and skills and is focussed on team leadership and decision making.5 The score sheet contains performance criteria that participants are expected to demonstrate in order to pass the test. Performance criteria are classified as bold (i.e. essential) and non-bold (desirable). Based on this structured assessment of performance the assessment outcome is reported as a single binary pass–fail mark. We have developed a four point scoring system which can be applied to each of the 24 performance criteria in order to characterise the quality of candidate performance in more detail. The scoring system does not give differential score for bold and non-bold treatment points.

The aim of this study is to determine the measurement properties of the new tool by considering the tool's internal structure and relationship with other variables (i.e. theoretical (post-course MCQ paper) and practical skill assessments) and thus report its utility to better characterise performance in CASTest than a simple pass/fail result.

Section snippets

Participants

The study was approved by South Birmingham Research Ethics Committee. Participants enrolled in an evaluation of pre-course computer simulation material from ALS courses at 11 UK centres were eligible for inclusion in the study. Participants provided written informed consent. The study was conducted between March and December 2007.

Assessment criteria

Participants underwent assessment of the following aspects: pre-/post-course MCQ paper, skills assessments (airway, initial assessment and resuscitation), and a

Results

A total of 537 participants were assessed: 346 doctors, 97 nurses, 7 operating department practitioners, 8 others and 79 unknown. There were more than double the number of junior staff compared to senior staff (267 vs 114), whilst only 27 were students and 129 unknown. Data for pre-course MCQ were available from 429 (79.9%) of participants.

Discussion

This study presents evidence supporting the validity of a scoring system which can be used to measure overall performance during advanced life support cardiac arrest simulation testing. This system will allow greater precision in classifying performance than the existing binary pass or fail outcome measure. Using a large cohort of multi-professional health care providers the participants awarded a pass mark by the standard assessment tool obtained significantly higher performance scores than

Conclusions

Evidence supporting the construct validity of the CASTest scoring system is presented. This simple scoring system better characterises performance in the ALS course CASTest than the current binary pass–fail outcome.

Conflict of interest statement

Gavin Perkins, Andy Lockey, Ian Bullock and Robin Davies are non-paid members of the Resuscitation Council UK ALS committee responsible for the development of the ALS course and its assessments.

Robin Davies is employed by the Resuscitation Council (UK).

Acknowledgments

We would like to thank the Resuscitation Research Group for assistance with the project: Nicole Gomez-Davis, Marie Fletcher, Teresa Melody, Fang Gao and Dawn Hill.

GDP holds a DH (NIHR) Clinician Scientist and is supported by the Intensive Care Foundation. The project was funded by the Resuscitation Council (UK) and Laerdal Foundation for Acute Medicine. The study design, analysis and decision to publish were the responsibility of the authors.

Collaborators: Julie Lawrence, Good Hope Hospital,

References (14)

There are more references available in the full text version of this article.

Cited by (34)

  • Advanced life support provider course in Italy: A 5-year nationwide study to identify the determinants of course success

    2015, Resuscitation
    Citation Excerpt :

    Accordingly, in our study, being an MD and having a high pre-course MCQ score were independent predictors of course success. It is reasonable to expect that clinical experience might improve knowledge and the levels of resuscitation competence.19,22,23 Indeed, nurses and doctors working in high-risk areas for cardiac arrest or encountering more cardiac arrest events have demonstrated greater BLS and ALS theoretical knowledge compared to their colleagues working in lesser exposed areas.7,21,24

View all citing articles on Scopus

A Spanish translated version of the abstract of this article appears as Appendix in the final online version at doi:10.1016/j.resuscitation.2009.04.043.

View full text