Weighting checklist items and station components on a large-scale OSCE: is it worth the effort?

Abstract:

BACKGROUND:Past research suggests that the use of externally-applied scoring weights may not appreciably impact measurement qualities such as reliability or validity. Nonetheless, some credentialing boards and academic institutions apply differential scoring weights based on expert opinion about the relative importance of individual items or test components of Observed Structured Clinical Examinations (OSCEs). AIMS:To investigate the impact of simplified scoring models that make little to no use of differential weighting on the reliability of scores and decisions on a high stakes OSCE required for medical licensure in Canada. METHOD:We applied four different weighting models of various complexities to data from three administrations of the OSCE. We compared score reliability, pass/fail rates, correlations between the scores and classification decision accuracy and consistency across the models and administrations. RESULTS:Less complex weighting models yielded similar reliability and pass rates as the more complex weighting model. Minimal changes in candidates' pass/fail status were observed and there were strong and statistically significant correlations between the scores for all scoring models and administrations. Classification decision accuracy and consistency were very high and similar across the four scoring models. CONCLUSIONS:Adopting a simplified weighting scheme for this OSCE did not diminish its measurement qualities. Instead of developing complex weighting schemes, experts' time and effort could be better spent on other critical test development and assembly tasks with little to no compromise in the quality of scores and decisions on this high-stakes OSCE.

journal_name

Med Teach

journal_title

Medical teacher

authors

Sandilands DD,Gotzmann A,Roy M,Zumbo BD,De Champlain A

doi

10.3109/0142159X.2014.899687

subject

Has Abstract

pub_date

2014-07-01 00:00:00

pages

585-90

issue

7

eissn

0142-159X

issn

1466-187X

journal_volume

36

pub_type

杂志文章
  • Using questionnaires to determine whether medical graduates' career choice is determined by undergraduate or postgraduate experiences.

    abstract:BACKGROUND:There has been increased interest in the factors influencing the career choice of doctors. Feminization of the medical workforce, changing health care needs, reform of training has ensured that this is an important issue for workforce planners and educationists. AIMS AND METHODS:Questionnaires were distribu...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/01421590701551755

    authors: Watmough S,Taylor D,Ryland I

    更新日期:2007-10-01 00:00:00

  • The clinical educator's guide to fostering learner motivation: AMEE Guide No. 137.

    abstract::Motivation theory and research remain underused by health professions educators. Some educators say it can seem too abstract. To address this, we applied health care language to learner motivation theories. Using a familiar metaphor, we examined the indications, mechanism of action, administration, and monitoring of l...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2020.1837764

    authors: Lyons KM,Cain JJ,Haines ST,Gasevic D,Brock TP

    更新日期:2020-11-02 00:00:00

  • Applying the Bookmark method to medical education: standard setting for an aseptic technique station.

    abstract:INTRODUCTION:The purpose was to evaluate the Bookmark standard-setting method for use on a performance-based assessment in medical education. METHODS:We compared cutscores for Aseptic performance assessment using the modified Angoff, Hofstee and modified Bookmark methods. RESULTS:The Angoff produced a cutscore of 62%...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2013.778395

    authors: Lypson ML,Downing SM,Gruppen LD,Yudkowsky R

    更新日期:2013-07-01 00:00:00

  • The role of academic health centers in addressing social responsibility.

    abstract::Despite the many successes achieved by academic health centers and the significant attention paid to the importance of the impact of social determinants on health, a broader movement of the academic health center community to share best practices and standardize these efforts across institutions and communities has no...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2011.590249

    authors: Wartman SA,Steinberg MJ

    更新日期:2011-01-01 00:00:00

  • Peer assisted learning in surgical skills laboratory training: a pilot study.

    abstract:BACKGROUND:Peer assisted learning (PAL) has been rarely investigated in surgical skills training. AIMS:Describe feedback residents give peers on surgical skills with and without guidelines, determine the association between feedback and actual performance, evaluate resident satisfaction with PAL. METHOD:Participants ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2012.706340

    authors: Beard JH,O'sullivan P,Palmer BJ,Qiu M,Kim EH

    更新日期:2012-01-01 00:00:00

  • Development and evaluation of cesarean section surgical training using computer-enhanced visual learning.

    abstract:BACKGROUND:Skilled performance of cesarean deliveries is essential in obstetrics and gynecology residency. A computer-enhanced visual learning module (CEVL Cesarean) was developed to teach cesarean deliveries. METHODS:An online module presented cesarean deliveries as a series of components using text, audio, video and...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2014.917156

    authors: York SL,Maizels M,Cohen E,Stoltz RS,Jamil A,McGaghie WC,Gossett DR

    更新日期:2014-11-01 00:00:00

  • Long-term benefits by a mind-body medicine skills course on perceived stress and empathy among medical and nursing students.

    abstract:BACKGROUND:A significant number of medical students suffer from burnout symptoms and reduced empathy. This controlled, quasi-experimental study aimed to investigate whether a mind-body medicine (MBM) skills course could reduce perceived stress and increase empathy and self-reflection in medical and nursing students. M...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2017.1309374

    authors: van Vliet M,Jong M,Jong MC

    更新日期:2017-07-01 00:00:00

  • Teaching problem solving and decision making in undergraduate medical education: an instructional strategy.

    abstract::Actual changes in the context of real care along with their impact on current practice modes prompted the authors to develop a new instructional strategy, which aims to improve the diagnostic reasoning competence of medical undergraduates. In this strategy the use of visual representations is promoted. A diagnostic pa...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159031000137490

    authors: Van Puymbroeck H,Remmen R,Denekens J,Scherpbier A,Bisoffi Z,Van Den Ende J

    更新日期:2003-09-01 00:00:00

  • The effect of educational games on medical students' learning outcomes: a systematic review: BEME Guide No 14.

    abstract:BACKGROUND:An educational game is 'an instructional method requiring the learner to participate in a competitive activity with preset rules.' A number of studies have suggested beneficial effects of educational games in medical education. AIM:The objective of this study was to systematically review the effect of educa...

    journal_title:Medical teacher

    pub_type: 杂志文章,评审

    doi:10.3109/01421590903473969

    authors: Akl EA,Pretorius RW,Sackett K,Erdley WS,Bhoopathi PS,Alfarah Z,Schünemann HJ

    更新日期:2010-01-01 00:00:00

  • Twelve tips for the introduction of emotional intelligence in medical education.

    abstract::Emotional intelligence (EI) is the ability to recognize, understand, and manage emotions in yourself and in others. EI has long been recognized as a critical component for individual and organizational success within the business realm, and there is emerging evidence that enhancing EI is equally important in the medic...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2018.1481499

    authors: Roth CG,Eldin KW,Padmanabhan V,Friedman EM

    更新日期:2019-07-01 00:00:00

  • How does medical education affect empathy and compassion in medical students? A meta-ethnography: BEME Guide No. 57.

    abstract::Background: Empathy and compassion are important in healthcare delivery, and are necessary qualities in medical students. Aims: To explore medical students', patients' and educators' perceptions of what affects empathy and the expression of compassion; and to address gaps in knowledge, attitudes and skills on how educ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2019.1630731

    authors: Krishnasamy C,Ong SY,Loo ME,Thistlethwaite J

    更新日期:2019-11-01 00:00:00

  • Connectivism: A knowledge learning theory for the digital age?

    abstract:BACKGROUND:The emergence of the internet, particularly Web 2.0 has provided access to the views and opinions of a wide range of individuals opening up opportunities for new forms of communication and knowledge formation. Previous ways of navigating and filtering available information are likely to prove ineffective in ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2016.1173661

    authors: Goldie JG

    更新日期:2016-10-01 00:00:00

  • Development of cognitive processing and judgments of knowledge in medical students: Analysis of progress test results.

    abstract:BACKGROUND:Beside acquiring knowledge, medical students should also develop the ability to apply and reflect on it, requiring higher-order cognitive processing. Ideally, students should have reached higher-order cognitive processing when they enter the clinical program. Whether this is the case, is unknown. We investig...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2016.1170781

    authors: Cecilio-Fernandes D,Kerdijk W,Jaarsma AD,Tio RA

    更新日期:2016-11-01 00:00:00

  • Learner preferences regarding integrating, sequencing and aligning virtual patients with other activities in the undergraduate medical curriculum: A focus group study.

    abstract:CONTEXT:E-learning resources, such as virtual patients (VPs), can be more effective when they are integrated in the curriculum. To gain insights that can inform guidelines for the curricular integration of VPs, we explored students' perceptions of scenarios with integrated and non-integrated VPs aimed at promoting clin...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2013.826790

    authors: Huwendiek S,Duncker C,Reichert F,De Leng BA,Dolmans D,van der Vleuten CP,Haag M,Hoffmann GF,Tönshoff B

    更新日期:2013-11-01 00:00:00

  • Community-based education: The influence of role modeling on career choice and practice location.

    abstract:INTRODUCTION:Research findings in medical education support the importance of positive role models in enhancing learning and influencing the career path of medical students and graduates. The authors explored the characteristics of positive and negative role models during Community-Based Education and Service (COBES), ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2016.1246711

    authors: Amalba A,Abantanga FA,Scherpbier AJ,van Mook WN

    更新日期:2017-02-01 00:00:00

  • Accreditation of specialty residency programmes in Canada.

    abstract::The accreditation process gives assurance to a number of individuals and groups about the quality of approved programmes. It is based on principles that have evolved over time. The process of the Royal College of Physicians and Surgeons of Canada is based upon a set of nine explicitly stated standards, each amplified ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/01421598909146281

    authors: Maudsley RF

    更新日期:1989-01-01 00:00:00

  • Comparability of outcome frameworks in medical education: Implications for framework development.

    abstract:BACKGROUND:Given the increasing mobility of medical students and practitioners, there is a growing need for harmonization of medical education and qualifications. Although several initiatives have sought to compare national outcome frameworks, this task has proven a challenge. Drawing on an analysis of existing outcome...

    journal_title:Medical teacher

    pub_type: 杂志文章,评审

    doi:10.3109/0142159X.2015.1012490

    authors: Hautz SC,Hautz WE,Feufel MA,Spies CD

    更新日期:2015-01-01 00:00:00

  • Medical education and the GCSE generation--a transferable activity-skills programme in medical sociology.

    abstract::The first major intake of medical students who studied the new GCSE occurred in Autumn 1990. These students have experienced a range of student-centred educational approaches to learning, but medical education in the UK remains committed to teacher-centred approaches. This mismatch of expectations is discussed. An inn...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/01421599109029029

    authors: Fox NJ,Dryden C

    更新日期:1991-01-01 00:00:00

  • Introducing medical educators to qualitative study design: Twelve tips from inception to completion.

    abstract::Many research questions posed by medical educators could be answered more effectively by the application of carefully selected qualitative research design than traditional quantitative research methods. Indeed, in many cases using mixed methods research would expand the scope of a study and yield meaningful qualitativ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2015.1035244

    authors: Ramani S,Mann K

    更新日期:2016-05-01 00:00:00

  • Curriculum Navigator: aspiring towards a comprehensive package for curriculum planning.

    abstract:BACKGROUND:Insightful frameworks for curriculum development were described in the literature. There is a need, however, to outline the approach we prefer, sometime unconsciously, in curriculum planning. AIMS:This article describes a novel conceptual framework called Curriculum Navigator, to explore our attitudes towar...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2012.689445

    authors: Al-Eraky MM

    更新日期:2012-01-01 00:00:00

  • Students' reflections in a portfolio pilot: highlighting professional issues.

    abstract:BACKGROUND:Portfolios are highlighted as potential assessment tools for professional competence. Although students' self-reflections are considered to be central in the portfolio, the content of reflections in practice-based portfolios is seldom analysed. AIM:To investigate whether students' reflections include suffic...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2010.509420

    authors: Haffling AC,Beckman A,Pahlmblad A,Edgren G

    更新日期:2010-01-01 00:00:00

  • How To…: Use Prepared TV Material and Films in Medical Education.

    abstract::The use of educational television requires special skills on the part of both teacher and student. The experience and research results of the Open University are used as a basis to describe some functions and associated pedagogical considerations of the use of television in medical education. Emphasis is placed on pre...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/01421598009071283

    authors: Durbridge N,Gale J

    更新日期:1980-01-01 00:00:00

  • Steering the patient mix of GP trainees: results of a randomized controlled intervention.

    abstract:BACKGROUND:In studies exploring the patient mixes of general practitioner (GP) trainees, gaps were repeatedly found, as there were disparities between the patient mixes of GP trainers and trainees. This reduces the opportunities of trainees to acquire enough competence. AIMS:To investigate whether steering the patient...

    journal_title:Medical teacher

    pub_type: 杂志文章,随机对照试验

    doi:10.3109/0142159X.2013.759197

    authors: de Jong J,Visser MR,Wieringa-de Waard M

    更新日期:2013-01-01 00:00:00

  • Current trends and future implications in the developing role of clinical skills centres.

    abstract::Clinical skills centres (CSCs) have contributed significantly to undergraduate medical education. Using alternative venues for training in communication skills, clinical examination and practical procedures they avoid jeopardising patient-care in traditional clinical settings. Using simulation they enable tutors to st...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/01421590120075724

    authors: Dent JA

    更新日期:2001-09-01 00:00:00

  • Expanding the reach of health sciences education and empowering others: the OpenCourseWare initiative at Tufts University.

    abstract::OpenCourseWare (OCW) represents an innovative and cost-effective opportunity for institutions to take a more active role in strengthening health sciences education worldwide. OCW content can provide a supplement to curricula available in resource-rich settings, as well as provide much of the basic content critical to ...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/01421590701881665

    authors: Lee MY,Albright S,O'Leary L,Terkla DG,Wilson N

    更新日期:2008-01-01 00:00:00

  • How we developed a comprehensive resuscitation-based simulation curriculum in emergency medicine.

    abstract::Over the past decade, simulation-based education has emerged as a new and exciting adjunct to traditional bedside teaching and learning. Simulation-based education seems particularly relevant to emergency medicine training where residents have to master a very broad skill set, and may not have sufficient real clinical...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2014.976187

    authors: Dagnone JD,McGraw R,Howes D,Messenger D,Bruder E,Hall A,Chaplin T,Szulewski A,Kaul T,O'Brien T

    更新日期:2016-01-01 00:00:00

  • Do GP trainers use feedback in drawing up their Personal Development Plans (PDPs)? Results from a quantitative study.

    abstract:BACKGROUND:General practice (GP) trainers play a key role GP trainees' education. To stimulate development of trainer competencies a Personal Development Plan (PDP) can be helpful, especially when feedback is incorporated. AIMS:To investigate to what extent GP trainers use feedback in PDPs. METHODS:GP trainers were p...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/0142159X.2012.670327

    authors: van Es JM,Visser MR,Wieringa-de Waard M

    更新日期:2012-01-01 00:00:00

  • Cross-cultural use and development of virtual patients.

    abstract::Three major issues drive the cross-cultural use of virtual patients (VPs): an increased mobility of healthcare professionals, students and patients; limited resources for developing VPs; and emerging standards for the exchange of VPs across institutions. Many students are trained in countries other than where they wer...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/01421590903124724

    authors: Fors UG,Muntean V,Botezatu M,Zary N

    更新日期:2009-08-01 00:00:00

  • Differences between medical school and PGY1 learning outcomes: An explanation for new graduates not being "work ready"?

    abstract::Background: Widespread concerns about new medical graduates' 'work readiness' may reflect, in part, differences in mandatory learning outcomes for medical students and new medical graduates.Purpose: To examine differences between required medical student and PGY1 (first year resident) training program outcomes, and th...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.1080/0142159X.2020.1782865

    authors: Ellis PM,Wilkinson TJ,Hu WC

    更新日期:2020-09-01 00:00:00

  • Choosing and designing knowledge assessments: experience at a new medical school.

    abstract:BACKGROUND:Curriculum developers have a wide choice of assessment methods in all aspects of medical education including the specific area of medical knowledge. When selecting the appropriate tool, there is an increasing literature to provide a robust evidence base for developments or decisions. AIM:As a new medical sc...

    journal_title:Medical teacher

    pub_type: 杂志文章

    doi:10.3109/01421591003614858

    authors: Freeman AC,Ricketts C

    更新日期:2010-01-01 00:00:00