The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE

ObjectiveAlthough previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills.MethodsThe University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training.ResultsFor analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales.ConclusionA combined analytic-holistic rating scale increased score reliability and was sensitive to level of training.Practice implicationsGiven increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.     

True communication skills assessment in interdepartmental OSCE stations: Standard setting using the MAAS-Global and EduG

Background

Comparing outcome of clinical skills assessment is challenging. This study proposes reliable and valid comparison of communication skills (1) assessment as practiced in Objective Structured Clinical Examinations (2). The aim of the present study is to compare CS assessment, as standardized according to the MAAS Global, between stations in a single undergraduate medical year.

Assessment of patient communication skills during OSCE: examining effectiveness of a training program in minimizing inter-grader variability

Objective

To assess effectiveness of a training program in reducing inter-grader variability in grading communication skills during an objective structured clinical exam (OSCE).

Methods

Global communication (GC) skills are assessed by standardized participants (SP) and faculty during each OSCE using a 6 item rubric. Despite criteria delineated in the GC rubric, inter-grader variability was observed. During 2008-2009 academic year, a training program was implemented before each OSCE to achieve more consistent interpretation and grading in GC skills. GC grades between SP and faculty for 2nd and 3rd level student OSCEs during 2008-2009 (post-training) were compared to 2007-2008 (pre-training). Data was analyzed using repeated measures ANOVA.

Results

274 and 281 students participated in OSCEs during 2007-2008 and 2008-2009 academic years, respectively. Training significantly (P < .001) decreased grader variability between SPs and faculty. There was a greater mean difference between faculty vs. SP before training (faculty 14.68, SP 15.87) compared to after training (faculty 13.51, SP 13.78). Mean GC scores for both faculty and SPs also decreased significantly after training.

Conclusion and practice implications

A training program may be necessary to reduce inter-rater variability in assessment of OSCE communication skills if it is to be truly helpful to student pharmacists learning to counsel patients.     

Optimizing the utility of communication OSCEs: omit station-specific checklists and provide students with narrative feedback

Objective

To evaluate how the utility (reliability, validity, acceptability, feasibility, cost and educational impact) of a communication-OSCE was influenced by whether or not station-specific (StSp) checklists were used together with a generic instrument and whether or not narrative feedback was provided to students.

Methods

At ten stations, faculty members rated standardized patient–student interactions using the Common Ground (CG) instrument (at all stations) and StSp-checklists. Both raters and patients provided written feedback. The impact of changing the design on the various utility parameters was assessed: reliability by means of a generalizability study, cost using the Reznick model and the other utility parameters by means of a survey.

Results

Use of the generic instrument (CG) proved more reliable (G coefficient = 0.67) than using the StSp-checklists (G = 0.47) or both (G = 0.65) while there was a high correlation between both scale scores (Pearsons’ r = 0.86). The cost was 6.5% higher when StSp-checklists were used and 5% higher when narrative feedback was provided.

Conclusion

The utility of a communication OSCE can be enhanced by omitting StSp-checklists and by providing narrative feedback to students.

Practice implications

The same generic assessment scale can be used in all stations of a communication OSCE. Providing feedback to students is promising but it increases the costs.     

Evaluating verbal and non-verbal communication skills, in an ethnogeriatric OSCE

Objective

Communication during medical interviews plays a large role in patient adherence, satisfaction with care, and health outcomes. Both verbal and non-verbal communication (NVC) skills are central to the development of rapport between patients and healthcare professionals. The purpose of this study was to assess the role of non-verbal and verbal communication skills on evaluations by standardized patients during an ethnogeriatric Objective Structured Clinical Examination (OSCE).

Methods

Interviews from 19 medical students, residents, and fellows in an ethnogeriatric OSCE were analyzed. Each interview was videotaped and evaluated on a 14 item verbal and an 8 item non-verbal communication checklist. The relationship between verbal and non-verbal communication skills on interview evaluations by standardized patients were examined using correlational analyses.

Results

Maintaining adequate facial expression (FE), using affirmative gestures (AG), and limiting both unpurposive movements (UM) and hand gestures (HG) had a significant positive effect on perception of interview quality during this OSCE. Non-verbal communication skills played a role in perception of overall interview quality as well as perception of culturally competent communication.

Conclusion

Incorporating formative and summative evaluation of both verbal and non-verbal communication skills may be a critical component of curricular innovations in ethnogeriatrics, such as the OSCE.     

The reliability of a modified Kalamazoo Consensus Statement Checklist for assessing the communication skills of multidisciplinary clinicians in the simulated environment

Objective

With increased recognition of the importance of sound communication skills and communication skills education, reliable assessment tools are essential. This study reports on the psychometric properties of an assessment tool based on the Kalamazoo Consensus Statement Essential Elements Communication Checklist.

Methods

The Gap-Kalamazoo Communication Skills Assessment Form (GKCSAF), a modified version of an existing communication skills assessment tool, the Kalamazoo Essential Elements Communication Checklist-Adapted, was used to assess learners in a multidisciplinary, simulation-based communication skills educational program using multiple raters. 118 simulated conversations were available for analysis. Internal consistency and inter-rater reliability were determined by calculating a Cronbach's alpha score and intra-class correlation coefficients (ICC), respectively.

Results

The GKCSAF demonstrated high internal consistency with a Cronbach's alpha score of 0.844 (faculty raters) and 0.880 (peer observer raters), and high inter-rater reliability with an ICC of 0.830 (faculty raters) and 0.89 (peer observer raters).

Conclusion

The Gap-Kalamazoo Communication Skills Assessment Form is a reliable method of assessing the communication skills of multidisciplinary learners using multi-rater methods within the learning environment.

Practice implications

The Gap-Kalamazoo Communication Skills Assessment Form can be used by educational programs that wish to implement a reliable assessment and feedback system for a variety of learners.     

Reliability of narrative assessment data on communication skills in a summative OSCE

ObjectiveTo quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation.MethodsNarrative comments and communication scores (scale 1–5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method.ResultsA total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5.ConclusionReliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently.Practice implicationsNarratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores.     

The Medical Examiner: reviewing all non-coronial deaths in England and Wales

The role of Medical Examiner was originally introduced in England and Wales by the Coroners and Justice Act 2009. A key aim is to ensure that concerns about the care of the deceased by families and whistle-blowers are identified and acted upon, to avoid repeated community and hospital-based care scandals. Since 2019 all acute hospital trusts have established their own Medical Examiner Service which is tailored to local needs and systems under the guidance of the National Medical Examiner. Medical Examiners are independent, senior, registered medical practitioners whose role is to ensure accuracy of the Medical Certificate of Cause of Death, ensure appropriate notification of deaths to His Majesty's Coroners and determine if there are any clinical governance concerns. This is achieved by reviewing the deceased's medical record; speaking to the attending doctor; and speaking to the next of kin. Medical Examiner Officers provide administrative support, service continuity and undertake many Medical Examiner functions by delegated authority. The majority of in-hospital deaths are now scrutinised and the process is rolling out to include all community deaths. It is expected that the process will become statutory by 2024 when every death within England and Wales will be reviewed either by the Medical Examiner or HM Coroner.     

Assessing communication skills in student pharmacists—Psychometric validation of Global Communication Rubric

Assessing communication skills in student pharmacistsPsychometric validation of the Global Communication Rubric.ObjectiveThe objective of this study was to perform psychometric validation of the Global Communication (GC) rubric.MethodsThe GC rubric was developed and used for the past 10 years at the institution to assess level of communication skills in Objective Structured Clinical Examinations (OSCE) in the PharmD program. The rubric consisted of 6 questions with the scoring criteria of 0 (failure), 1 (needs improvement), 2 (satisfactory), and 3 (excellent). Data from GC rubric scores for Classes 2010 to 2020 (1,100 students) was used to perform psychometric validation by assessing the following properties: scale reliability; face, content, construct, and predictive validity; and responsiveness.ResultsInternal consistency reliability was acceptable at Cronbach’s alpha of 0.78. Construct validity showed that all six items loaded highly onto one component. Responsiveness was demonstrated using uptrend of the score improvement as the students advance in the curriculum from P1 to P3 levels.ConclusionThe result of this study confirmed reliability and validity of the GC rubric in evaluating communication skills of student pharmacists.Practice implicationsThe GC rubric is valuable in evaluating short encounters in an easy and expedited manner.     

Outcome of parent-physician communication skills training for pediatric residents

Objective

Communication skills represent an essential component of clinical competence. In the field of pediatrics, communication between physicians and patients’ parents is characterized by particular difficulties. To investigate the effects of a parent-physician communication skills training program on OSCE performance and self-efficacy in a group control design.

Methods

Parallel to their daily work in the outpatient department, intervention-group experienced clinicians in practice (n = 14) participated in a communication training with standardized parents. Control-group physicians (n = 14) did not receive any training beyond their daily work. Performance was assessed by independent video ratings of an OSCE. Both groups rated their self-efficacy prior to and following training.

Results

Regarding OSCE performance, the intervention group demonstrated superior skills in building relationships with parents (p < .024) and tended to perform better in exploring parents’ problems (p < .081). The communication training program led to significant improvement in self-efficacy with respect to the specific training objectives in the intervention group (p < .046).

Conclusion

Even in physicians with considerable experience, structured communication training with standardized parents leads to significant improvement in OSCE performance and self-efficacy.

Practise implications

Briefness and tight structure make the presented communication training program applicable even for experienced physicians in daily clinical practice.     

The feasibility of a multi-format Web-based assessment of physicians' communication skills

Objective

Little is known about the best approaches and format for measuring physicians’ communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats.

Methods

We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources.

Results

Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0).

Conclusion

The acceptable reliability and participants’ overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment.

Practice implications

With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians’ communication skills.     

Validación de la versión en español de un instrumento de evaluación de la comunicación centrada en el paciente en OSCE

ObjectiveThe aim of this study was to validate the Spanish version of the Communication Assessment Tool (CAT) to be used in an Objective Structured Clinical Evaluation (OSCE) to assess communication skills.MethodsData were collected from standardized patients evaluating 125 recent medical school graduates during a 12-station OSCE. A total of 1,500 completed CAT instruments were analyzed. Exploratory and confirmatory factorial analysis were performed and tested the internal consistency of the tool.ResultsWe found a single factor solution in the Spanish version of the instrument, similar to that described by the original author. Cronbach's alpha was high, ranging from 0.93 to 0.97. The overall mean percentage of ‘excellent’ ratings for the recent graduates was 38.07%.ConclusionsThe Spanish version of CAT is a feasible tool to be used in OSCE by standardized patients to assess communication skills in a standardized manner.     

General principles to consider when designing a clinical communication assessment program

ObjectivesAssessment of clinical communication helps teachers in healthcare education determine whether their learners have acquired sufficient skills to meet the demands of clinical practice. The aim of this paper is to give input to educators when planning how to incorporate assessment into clinical communication teaching by building on the authors’ experience and current literature.MethodsA summary of the relevant literature within healthcare education is discussed, focusing on what and where to assess, how to implement assessment and how to choose appropriate methodology.ResultsEstablishing a coherent approach to teaching, training, and assessment, including assessing communication in the clinical context, is discussed. Key features of how to implement assessment are presented including: establishing a system with both formative and summative approaches, providing feedback that enhances learning and establishing a multi-source and longitudinal assessment program.ConclusionsThe implementation of a reliable, valid, credible, feasible assessment method with specific educational relevance is essential for clinical communication teaching.Practice implicationsAll assessment methods have strengths and limitations. Since assessment drives learning, assessment should be aligned with the purpose of the teaching program. Combining the use of different assessment formats, multiple observations, and independent measurements in different settings is advised.     

Expert consensus on gender criteria for assessment in medical communication education

Objective

The aim of this study is to develop gender criteria that can be included in communication skills assessment in medical education.

Methods

A three-round Delphi study was conducted. The invited 59 participants were experts in the field of gender medicine education (n = 28) and doctor–patient communication (n = 31). Each Delphi round comprised a questionnaire, an analysis, and a feedback report. In the first round, gender experts explored gender themes in doctor–patient communication from which initial gender criteria were defined. The second and third rounds were used to validate the importance and feasibility of gender criteria. Consensus was defined as a 75% panel agreement and a mean of 4 or higher on a 5-point Likert scale.

Results

Four gender criteria achieved consensus after the third round. The importance of including the gender criteria in communication skills assessment was rated consistently higher than its feasibility. Gender criteria relating to the patients’ perspective, to gathering information and to gender and power were considered the most important.

Conclusion

Using a Delphi study, we have developed gender criteria for inclusion in communication skills assessment to promote good communication between doctors and patients.

Practice implications

Gender influences medical communication. Incorporating gender in communication skills assessment may be useful to improve the teaching and learning of communication skills.     

临床医学专业“七转八”入学考试增加客观结构化临床考试实践探索

目的招收优秀的临床医学七年制医学生进入北京协和医学院临床医学八年制"七转八"项目学习。方法在临床医学专业"七转八"项目入学考试中增加5个考站的客观结构化临床考试(OSCE)。结果对考试成绩进行分析显示,OSCE的总成绩、笔试总分和沟通交流等项目近乎于正态分布,测试能力较好;SP问诊、查体和英语笔试没能区分出差异。考生的笔试成绩与各考站的相关系数在0.048~0.334之间。结论 2014年"七转八"入学OSCE项目的设计是成功的,有必要在"七转八"入学考试增加OSCE。     

The effects of scenario-based communication training on nurses’ communication competence and self-efficacy and myocardial infarction knowledge

Objective

The aim of the study was to determine the effects of a simulated communication training course on nurses’ communication competence, self-efficacy, communication performance, myocardial infarction knowledge, and general satisfaction with their learning experience.

Methods

A randomized controlled trial was conducted with a pre-test and two post-tests. The experimental group underwent simulated communication training course and the control group received a case-based communication training course.

Results

The experimental group made more significant improvement in competence and self-efficacy in communication from pre-test to the second post-test than the control group. Although both groups’ satisfaction with their learning experience significantly increased from the first post-test to the second post-test, the experimental group was found to be more satisfied with their learning experience than the control group. No significant differences in communication performance and myocardial infarction knowledge between the two groups were identified.

Conclusion

Scenario-based communication training can be more fully incorporated into in-service education for nurses to boost their competence and self-efficacy in communication and enhance their communication performance in myocardial infarction patient care.

Practice implications

Introduction of real-life communication scenarios through multimedia in communication education could make learners more motivated to practice communication, hence leading to improved communication capacity.     

Developing communication curricula in healthcare education: An evidence-based guide

ObjectiveTo present a guide for communication curriculum development in healthcare professions for educators and curriculum planners.MethodsWe collated a selection of theories, frameworks and approaches to communication curriculum development to provide a roadmap of the main factors to consider when developing or enhancing communication skills curricula.ResultsWe present an evidence-based guide for developing and enhancing communication curriculum that can be applied to undergraduate and postgraduate healthcare education. Recommended steps to consider during the communication curricula development process include thoughtful examination of current communication education, needs assessment, focused learning goals and objectives, incorporation of experiential educational strategies allowing for skills practice and feedback and use of formative and summative assessment methods. A longitudinal, developmental and helical implementation approach contributes to reinforcement and sustainment of learners’ knowledge and skills.Conclusion and practice implicationsDrawing on best practices in developing communication curricula can be helpful in ensuring successful approaches to communication skills training for any level of learner or healthcare profession. This position paper provides a guide and identifies resources for new and established communication curriculum developers to reflect on strengths and opportunities in their own approaches to addressing the communication education needs of their learners.     

Assessment of surgeon communication skills from the patient perspective: A national evaluation using the Communication Assessment Tool

ObjectiveThe Communication Assessment Tool (CAT) has previously been translated and adapted to the Italian context. This national study aimed to validate the CAT and evaluate communication skills of practicing surgeons from the patient perspective.MethodsCAT consists of 14 items associated with a 5-point scale (5 = excellent); results are reported as the percent of ‘‘excellent’’ scores. It was administered to 920 consenting outpatients aged 18–84 in 26 Italian surgical departments.ResultsThe largest age group was 45–64 (43.8%); 52.2% of the sample was male. Scores ranged from 44.6% to 66.6% excellent. The highest-scoring items were “Treated me with respect” (66.6%), “Gave me as much information as I wanted” (66.3%) and “Talked in terms I could understand” (66.0%); the lowest was “Encouraged me to ask questions” (44.6%). Significant differences were associated with age (18–24 year old patients exhibited the lowest scores) and geographical location (Northern Italy had the highest scores).ConclusionCAT is a valid tool for measuring communication in surgical settings.Practice ImplicationsResults suggest that expectations of young people for communication in surgical settings are not being met. While there is room to improve communication skills of surgeons across Italy, patients highlighted the greatest need in the Central and Southern regions.     

Measuring patient views of physician communication skills: development and testing of the Communication Assessment Tool

OBJECTIVE: Interpersonal and communication skills have been identified as a core competency that must be demonstrated by physicians. We developed and tested a tool that can be used by patients to assess the interpersonal and communication skills of physicians-in-training and physicians-in-practice. METHODS: We began by engaging in a systematic scale development process to obtain a psychometrically sound Communication Assessment Tool (CAT). This process yielded a 15-item instrument that is written at the fourth grade reading level and employs a five-point response scale, with 5=excellent. Fourteen items focus on the physician and one targets the staff. Pilot testing established that the CAT differentiates between physicians who rated high or low on a separate satisfaction scale. We conducted a field test with physicians and patients from a variety of specialties and regions within the US to assess the feasibility of using the CAT in everyday practice. RESULTS: Thirty-eight physicians and 950 patients (25 patients per physician) participated in the field test. The average patient-reported mean score per physician was 4.68 across all CAT items (S.D.=0.54, range 3.97-4.95). The average proportion of excellent scores was 76.3% (S.D.=11.1, range 45.7-95.1%). Overall scale reliability was high (Cronbach's alpha=0.96); alpha coefficients were uniformly high when reliability was examined per doctor. CONCLUSION: The CAT is a reliable and valid instrument for measuring patient perceptions of physician performance in the area of interpersonal and communication skills. The field test demonstrated that the CAT can be successfully completed by both physicians and patients across clinical specialties. Reporting the proportion of "excellent" ratings given by patients is more useful than summarizing scores via means, which are highly skewed. PRACTICE IMPLICATIONS: Specialty boards, residency programs, medical schools, and practice plans may find the CAT valuable for both collecting information and providing feedback about interpersonal and communication skills.