Malau-Aduli, Bunmi S., Mulcahy, Sue, Warnecke, Emma, Otahal, Petr, Teague, Peta-Ann, Turner, Richard, and Van der Vleuten, Cees (2012) Inter-rater reliability: comparison of checklist and global scoring for OSCEs. Creative Education, 3 (6A). pp. 937-942.

PDF (Published Version) - Published Version Restricted to Repository staff only

Abstract

Objective Structured Clinical Examinations (OSCEs) have been used globally in evaluating clinical competence in the education of health professionals. Despite the objective intent of OSCEs, scoring methods used by examiners have been a potential source of measurement error affecting the precision with which test scores are determined. In this study, we investigated the differences in the inter-rater reliabilities of objective checklist and subjective global rating scores of examiners (who were exposed to an online training program to standardise scoring techniques) across two medical schools. Examiners' perceptions of the e-scoring program were also investigated. Two Australian universities shared three OSCE stations in their end-of-year undergraduate medical OSCEs. The scenarios were video-taped and used for on-line examiner training prior to actual exams. Examiner ratings of performance at both sites were analysed using generalisability theory. A single facet, all random persons by raters design [PxR] was used to measure inter-rater reliability for each station, separate for checklist scores and global ratings. The resulting variance components were pooled across stations and examination sites. Decision studies were used to measure reliability estimates. There was no significant mean score difference between examination sites. Variation in examinee ability accounted for 68.3% of the total variance in checklist scores and 90.2% in global ratings. Rater contribution was 1.4% & 0% of the total variance in checklist score and global rating respectively, reflecting high inter-rater reliability of the scores provided by co-examiners across the two schools. Score variance due to interaction and residual error was larger for checklist scores (30.3% vs 9.7%) than for global ratings. Reproducibility coefficients for global ratings were higher than for checklist scores. Survey results showed that the e-scoring package facilitated consensus on scoring techniques. This approach to examiner training also allowed examiners to calibrate the OSCEs in their own time. This study revealed that inter-rater reliability was higher for global ratings than for checklist scores, thus providing further evidence for the reliability of subjective examiner ratings.

Item ID:	32505
Item Type:	Article (Scholarly Work)
ISSN:	2151-4771
Keywords:	objective structured clinical examination; inter-rater reliability; checklist scores; global ratings
Related URLs:	Publisher
Date Deposited:	27 Jun 2014 01:40
FoR Codes:	11 MEDICAL AND HEALTH SCIENCES > 1199 Other Medical and Health Sciences > 119999 Medical and Health Sciences not elsewhere classified @ 100%
SEO Codes:	92 HEALTH > 9299 Other Health > 929999 Health not elsewhere classified @ 80% 93 EDUCATION AND TRAINING > 9302 Teaching and Instruction > 930202 Teacher and Instructor Development @ 20%
Downloads:	Total: 2
	More Statistics