The Library

Design, validation and dissemination of an undergraduate assessment tool using SimMan® in simulated medical emergencies

Tools

Paskins, Zoe, Kirkcaldy, Jo, Allen, Maggie, Dr., Macdougall, Colin, Dr., Fraser, Ian, Dr. and Peile, Ed. (2010) Design, validation and dissemination of an undergraduate assessment tool using SimMan® in simulated medical emergencies. Medical Teacher, Volume 32 (Number 1). E12-E17. ISSN 0142-159X

Preview

Text
WRAP_Macdougall_Design validation etc final.pdf - Accepted Version
Download (492Kb) | Preview

Official URL: http://dx.doi.org/10.3109/01421590903199643

Abstract

Background: Increasingly, medical students are being taught acute medicine using whole-body simulator manikins.

Aim: We aimed to design, validate and make widely available two simple assessment tools to be used with Laerdal SimMan (R) for final year students.

Methods: We designed two scenarios with criterion-based checklists focused on assessment and management of two medical emergencies. Members of faculty critiqued the assessments for face validity and checklists revised. We assessed three groups of different experience levels: Foundation Year 2 doctors, third and final year medical students. Differences between groups were analysed, and internal consistency and interrater reliability calculated. A generalisability analysis was conducted using scenario and rater as facets in design.

Results: A maximum of two items were removed from either checklist following the initial survey. Significantly different scores for three groups of experience for both scenarios were reported (p<0.001). Interrater reliability was excellent (r>0.90). Internal consistency was poor (alpha<50.5). Generalizability study results suggest that four cases would provide reliable discrimination between final year students.

Conclusions: These assessments proved easy to administer and we have gone some way to demonstrating construct validity and reliability. We have made the material available on a simulator website to enable others to reproduce these assessments.

Item Type:

Journal Article

Alternative Title:

SimMan assessment tool : design and validation

Subjects:

L Education > LB Theory and practice of education
R Medicine > R Medicine (General)

Divisions:

Faculty of Social Sciences > Institute for Employment Research
Faculty of Medicine > Warwick Medical School

Library of Congress Subject Headings (LCSH):

Medicine -- Study and teaching (Higher) -- Simulation methods, Emergency medical services -- Study and teaching (Higher) -- Simulation methods, Simulated patients -- Design, Medical education, Physicians -- Training of -- Great Britain

Journal or Publication Title:

Medical Teacher

Publisher:

Informa Healthcare

ISSN:

0142-159X

Official Date:

January 2010

Dates:

Date	Event
January 2010	Published

Volume:

Volume 32

Number:

Number 1

Number of Pages:

Page Range:

E12-E17

Identification Number:

10.3109/01421590903199643

Status:

Peer Reviewed

Publication Status:

Published

Access rights to Published version:

Restricted or Subscription Access

References:

Beilock SL, Bertenthal BI, McLay AM, Carr TH. 2004. Haste does not
always make waste: Expertise, direction of attention, and speed versus
accuracy in performing psychomotor skills. Psychon Bull Rev
11:373–379.
Bond WF, Spillane L. 2002. The use of simulation for emergency medicine
resident assessment. Acad Emerg Med 9(11):1295–1299.
Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. 2003.
Reliability and validity of a simulation-based acute care skills assessment
for medical students and residents. Anesthesiology
99(6):1270–1280.
Bradley P. 2006. The history of simulation in medical education and
possible future directions. Med Educ 40(3):254–262.
Byrne AJ, Greaves JD. 2001. Assessment instruments used during anaesthetic
simulation: Review of published studies. Br J Anaesth 86(3):445–450.
Cronbach LJ. 2004. My current thoughts on coefficient alpha and successor
procedures. Educ Psychol Meas 64(3):391–418.
Cullinane M, Findlay G, Hargraves C, Lucas S. 2005. National Confidential
Enquiry into Patient Outcome and Death. An Acute Problem [Internet].
Available from http://www.ncepod.org.uk/2005report/. Accessed 25
October 2008.
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Noel AG, Szalai JP. 1998.
Testing internal consistency and construct validity during evaluation of
performance in a patient simulator. Anesth Analg 86(6):1160–1164.
Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. 2001. The validity of
performance assessments using simulation. Anesthesiology
95(1):36–42.
Franklin C, Mathew J. 2002. Developing strategies to prevent in-hospital
cardiac arrest: Analysing responses of physicians and nurses in the
hours before the event. Crit Care Med 22:244–247.
General Medical Council 2003. Tomorrow’s doctors. London: General
Medical Council.
Gordon JA, Tancredi DN, Binder WD, Wilkerson WM, Shaffer DW. 2003.
Assessment of a clinical performance evaluation tool for use in a
simulator-based testing environment: A pilot study. Acad Med 78(Suppl
10):S45–47.
Guilford JP. 1954. Psychometric methods. 2nd ed. New York: McGraw-Hill
Book Co.
Hodgetts TJ, Kenward G, Vlackonikolis I, Payne S, Castle N, Crouch R,
Ineson N, Shaikh L. 2002. Incidence, location and reasons for avoidable
in-hospital cardiac arrest in a district general hospital. Resuscitation
54:115–123.
MacDowall J. 2006. The assessment and treatment of the acutely ill
patient – The role of the patient simulator as a teaching tool in the
undergraduate programme. Med Teach 28(4):326–329.
Maran NJ, Glavin RJ. 2003. Low- to high-fidelity simulation – A continuum
of medical education? Med Educ 37(Suppl 1):22–28.
McQuillan P, Pilkington S, Allan A, Taylor B, Short A, Morgan G, Nielson M,
Barrett D, Smith G. 1998. Confidential inquiry into quality of care before
admission to intensive care. BMJ 316:1853–1858.
Miller GE. 1990. The assessment of clinical skills/competence/performance.
Acad Med 65(9):S63–67.
Morgan PJ, Cleave-Hogg D. 2000a. A Canadian simulation experience:
Faculty and student opinions of a performance evaluation study. Br J
Anaesth 85(5):779–781.
Morgan PJ, Cleave-Hogg D. 2000b. Evaluation of medical
students’ performance using the anaesthesia simulator. Med Educ
34(1):42–45.
Morgan PJ, Cleave-Hogg D, Guest GB. 2001a. A comparison of global
ratings and checklist scores from an undergraduate assessment using an
anesthesia simulator. Acad Med 76(10):1053–1055.
Morgan PJ, Cleave-Hogg DM, Guest CB, Herold J. 2001b. Validity and
reliability of undergraduate performance assessments in an anesthesia
simulator. Can J Anaesth 48(3):225–233.
Morgan PJ, Cleave-Hogg D, DeSousa S, Tarshis J. 2004. High-fidelity
patient simulation: Validation of performance checklists. Br J
Anaesth 92(3):388–392.
Murray D, Boulet J, Ziv A, Woodhouse J, Kras J, Mcallister J. 2002. An acute
care skills evaluation for graduating medical students: A pilot study
using clinical simulation. Med Educ 36(9):833–841.
Mushquash C, O’Connor BP. 2006. SPSS and SAS programs for generalizability
theory analyses. Behav Res Methods 38(3):542–547.
Perkins GD, Barret H, Bullock I, Gabbott DA, Nolan JP, Mitchell S, Short A,
Smith CM, Smith GB, Todd S, Bion JF. 2005. The Acute Care
Undergraduate TEaching (ACUTE) initiative: Consensus development
of core competencies in acute care for undergraduates in the United
Kingdom. Intensive Care Med 31(12):1627–1633.
Schuwirth LWT, Van der Vleuten CPM. 2003. The use of clinical simulations
in assessment. Med Educ 37(Suppl 1):65–71.
Weller J, Robinson B, Larsen P, Caldwell C. 2004. Simulation-based training
to improve acute care skills in medical undergraduates. N Z Med J
117(1204):U1119.