“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

Article


Sturman, Nancy, Ostini, Remo, Wong, Wai Yee, Zhang, Jianzhen and David, Michael. 2017. "“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study." BMC Medical Education. 17 (1), pp. 1-7. https://doi.org/10.1186/s12909-017-0929-9
Article Title

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

ERA Journal ID13446
Article CategoryArticle
AuthorsSturman, Nancy, Ostini, Remo, Wong, Wai Yee, Zhang, Jianzhen and David, Michael
Journal TitleBMC Medical Education
Journal Citation17 (1), pp. 1-7
Article Number101
Number of Pages7
Year06 Jun 2017
PublisherBioMed Central Ltd.
Place of PublicationUnited Kingdom
ISSN1472-6920
Digital Object Identifier (DOI)https://doi.org/10.1186/s12909-017-0929-9
Web Address (URL)https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-017-0929-9
Abstract

Background: Robust and defensible clinical assessments attempt to minimise differences in student grades which are due to differences in examiner severity (stringency and leniency). Unfortunately there is little evidence to date that examiner training and feedback interventions are effective; “physician raters” have indeed been deemed “impervious to feedback”. Our aim was to investigate the effectiveness of a general practitioner examiner feedback intervention, and explore examiner attitudes to this.

Methods: Sixteen examiners were provided with a written summary of all examiner ratings in medical student clinical case examinations over the preceding 18 months, enabling them to identify their own rating data and compare it with other examiners. Examiner ratings and examiner severity self-estimates were analysed pre and post intervention, using non-parametric bootstrapping, multivariable linear regression, intra-class correlation and Spearman’s correlation analyses. Examiners completed a survey exploring their perceptions of the usefulness and acceptability of the intervention, including what (if anything) examiners planned to do differently as a result of the feedback.

Results: Examiner severity self-estimates were relatively poorly correlated with measured severity on the two clinical case examination types pre-intervention (0.29 and 0.67) and were less accurate post-intervention. No significant effect of the intervention was identified, when differences in case difficulty were controlled for, although there were fewer outlier examiners post-intervention. Drift in examiner severity over time prior to the intervention was observed. Participants rated the intervention as interesting and useful, and survey comments indicated that fairness, reassurance, and understanding examiner colleagues are important to examiners.

Conclusions: Despite our participants being receptive to our feedback and wanting to be “on the same page”, we did not demonstrate effective use of the feedback to change their rating behaviours. Calibration of severity appears to be difficult for examiners, and further research into better ways of providing more effective feedback is indicated.

KeywordsAustralia; Calibration; Certification; Clinical Competence; Education, Medical, Undergraduate; Educational Measurement; Faculty; Formative Feedback; General Practice; Humans; Judgment; Observer Variation; Students, Medical
Byline AffiliationsUniversity of Queensland
Department of Health, Queensland
Permalink -

https://research.usq.edu.au/item/wv29v/-on-the-same-page-the-effect-of-gp-examiner-feedback-on-differences-in-rating-severity-in-clinical-assessments-a-pre-post-intervention-study

Download files


Published Version
On the same page.pdf
License: CC BY 4.0
File access level: Anyone

  • 16
    total views
  • 5
    total downloads
  • 1
    views this month
  • 1
    downloads this month

Export as

Related outputs

How and where physicians learn their craft are central to determining where their medical careers unfold
Ostini, Remo. 2021. "How and where physicians learn their craft are central to determining where their medical careers unfold." Medical Journal of Australia. 215, pp. S5-S6. https://doi.org/10.5694/mja2.51122
Health literacy in rural and urban populations: A systematic review
Aljassim, Noor and Ostini, Remo. 2020. "Health literacy in rural and urban populations: A systematic review." Patient Education and Counseling. 103 (10), pp. 2142-2154. https://doi.org/10.1016/j.pec.2020.06.007
Common mental disorders among Indigenous people living in regional, remote and metropolitan Australia: a cross-sectional study
Nasir, Bushra F., Toombs, Maree T., Kondalsamy-Chennakesavan, Srinivas, Kisely, Steve, Gill, Neeraj S., Black, Emma, Hayman, Noel, Ranmuthugala, Geetha, Beccaria, Gavin, Ostini, Remo and Nicholson, Geoffrey C.. 2018. "Common mental disorders among Indigenous people living in regional, remote and metropolitan Australia: a cross-sectional study." BMJ Open. 8 (6), pp. 1-11. https://doi.org/10.1136/bmjopen-2017-020196
Insights from the coalface: barriers to accessing medicines and pharmacy services for resettled refugees from Africa
Bellamy, Kim, Ostini, Remo, Martini, Nataly and Kairuz, Therese. 2019. "Insights from the coalface: barriers to accessing medicines and pharmacy services for resettled refugees from Africa." Australian Journal of Primary Health. 25 (2), pp. 118-124. https://doi.org/10.1071/PY18092
Validation of the Shortened Perceived Medical Condition Self-Management Scale in Patients with Chronic Disease
Wild, Marcus G., Ostini, Remo, Harrington, Magdalena, Cavanaugh, K. and Wallston, Kenneth A.. 2018. "Validation of the Shortened Perceived Medical Condition Self-Management Scale in Patients with Chronic Disease." Psychological Assessment. 30 (10), pp. 1300-1307. https://doi.org/10.1037/pas0000572
A retrospective observational study on the management of urinary tract infections in older medical patients in the hospital
Alwis, D. D., Arjunan, A., Lin, S. and Ostini, R.. 2018. "A retrospective observational study on the management of urinary tract infections in older medical patients in the hospital." Australian and New Zealand Society for Geriatric Medicine Annual Scientific Meeting (2018). Sydney, Australia 27 - 29 Jun 2018 Australia. https://doi.org/10.1111/ajag.12551
Medical student contact with specialty trainees: Missing out in general practice?
Sturman, Nancy and Ostini, Remo. 2018. "Medical student contact with specialty trainees: Missing out in general practice?" Australian Journal of General Practice. 47 (6), pp. 391-395. https://doi.org/10.31128/AJGP-12-17-4423
Mental Health and Indigenous Connection to Land and Community
Toombs, M., Kondalsamy-Chennakesavan, S., Kisely, S., Gill, N., Hayman, N., Ranmuthugala, G., Beccaria, G., Ostini, R., Nicholson, G. and Nasir, B.. 2018. "Mental Health and Indigenous Connection to Land and Community." Royal Australian & New Zealand College of Psychiatrists (RANZCP) 2018 Congress. New Zealand 12 201 - 17 May 2018 Australia. SAGE Publications Ltd.
The Cultural Appropriateness of the Structured Clinical Interview for DSM-IV TR, AXIS I for Indigenous Populations
Toombs, M, Nasir, B, Kisely, S, Kondalsamy-Chennakesavan, S, Gill, N, Black, E, Hayman, N, Ranmuthugala, G, Beccaria, G, Ostini, R and Nicholson, G. 2018. "The Cultural Appropriateness of the Structured Clinical Interview for DSM-IV TR, AXIS I for Indigenous Populations." Royal Australian & New Zealand College of Psychiatrists (RANZCP) 2018 Congress. New Zealand 12 201 - 17 May 2018 Australia. SAGE Publications Ltd. https://doi.org/10.1177/0004867418764980
An Integrated Theatre Production for School Nutrition Promotion Program
Bush, Robert, Capra, Sandra, Box, Selina, McCallum, David, Khalil, Stephanie and Ostini, Remo. 2018. "An Integrated Theatre Production for School Nutrition Promotion Program." Children. 5 (3), pp. 1-11. https://doi.org/10.3390/children5030035
Factors associated with the human papillomavirus (HPV) vaccination across three countries following vaccination introduction
Nickel, Brooke, Dodd, Rachael H., Turner, Robin M., Waller, Jo, Marlow, Laura, Zimet, Gregory, Ostini, Remo and McCaffery, Kirsten. 2017. "Factors associated with the human papillomavirus (HPV) vaccination across three countries following vaccination introduction." Preventive Medicine Reports. 8, pp. 169-176. https://doi.org/10.1016/j.pmedr.2017.10.005
Perspectives of resettled African refugees on accessing medicines and pharmacy services in Queensland, Australia
Bellamy, Kim, Ostini, Remo, Martini, Nataly and Kairuz, Therese. 2017. "Perspectives of resettled African refugees on accessing medicines and pharmacy services in Queensland, Australia." International Journal of Pharmacy Practice. 25 (5), pp. 358-364. https://doi.org/10.1111/ijpp.12324
Perceptions of responsibility for health literacy among health professionals and consumers
Ostini, Remo, Ostini, Jenny and Argall, Rosemary. 2016. "Perceptions of responsibility for health literacy among health professionals and consumers." 14th International Congress of Behavioral Medicine: Behavioral Medicine: Making an Impact in the Modern World (ICBM 2016). Melbourne, Australia 07 - 10 Dec 2016 Springer. https://doi.org/10.1007/s12529-016-9586-3
Extending and validating a human papillomavirus (HPV) knowledge measure in a national sample of Canadian parents of boys
Perez, Samara, Tatar, Ovidiu, Ostini, Remo, Shapiro, Waller, Jo, Zimet, Gregory and Rosberger, Zeev. 2016. "Extending and validating a human papillomavirus (HPV) knowledge measure in a national sample of Canadian parents of boys." Preventive Medicine. 91, pp. 43-49. https://doi.org/10.1016/j.ypmed.2016.07.017
Patient Assessment of Chronic Illness Care (PACIC) in Type 2 Diabetes: A Longitudinal Study
Aung, Eindra, Ostini, Remo, Dower, Jo, Donald, Maria, Coll, Joseph R., Williams, Gail M. and Doi, Suhail A. R.. 2016. "Patient Assessment of Chronic Illness Care (PACIC) in Type 2 Diabetes: A Longitudinal Study." Evaluation and the Health Professions. 39 (2), pp. 185-203. https://doi.org/10.1177/0163278714556674
Seeking to understand: using generic qualitative research to explore access to medicines and pharmacy services among resettled refugees
Bellamy, Kim, Ostini, Remo, Martini, Nataly and Kairuz, Therese. 2016. "Seeking to understand: using generic qualitative research to explore access to medicines and pharmacy services among resettled refugees." International Journal of Clinical Pharmacy. 38 (3), pp. 671-675. https://doi.org/10.1007/s11096-016-0261-1
Erratum to: Seeking to understand: using generic qualitative research to explore access to medicines and pharmacy services among resettled refugees
Bellamy, Kim, Ostini, Remo, Martini, Nataly and Kairuz, Therese. 2016. "Erratum to: Seeking to understand: using generic qualitative research to explore access to medicines and pharmacy services among resettled refugees." International Journal of Clinical Pharmacy. 38 (2), p. 469. https://doi.org/10.1007/s11096-016-0276-7
Differences in Moral Judgment on Animal and Human Ethics Issues between University Students in Animal-Related, Human Medical and Arts Programs
Verrinder, Joy M., Ostini, Remo and Phillips, Clive J. C.. 2016. "Differences in Moral Judgment on Animal and Human Ethics Issues between University Students in Animal-Related, Human Medical and Arts Programs." PLoS One. 11 (3), pp. 1-15. https://doi.org/10.1371/journal.pone.0149308
Exploratory Study of Total and Free Prednisolone Plasma Exposure and Cushingoid Appearance, Quality of Life and Biochemical Toxicity in Adult Male Kidney Transplant Recipients
Bergmann, Troels K., Isbel, Nicole M., Ostini, Remo, Barraclough, Katherine A., Campbell, Scott B., McWhinney, Brett C., Inder, Warrick J., Russell, Anthony and Staatz, Christine E.. 2015. "Exploratory Study of Total and Free Prednisolone Plasma Exposure and Cushingoid Appearance, Quality of Life and Biochemical Toxicity in Adult Male Kidney Transplant Recipients." Clinical Drug Investigation. 35 (11), pp. 743-750. https://doi.org/10.1007/s40261-015-0334-2
Health literacy among consumers in community pharmacy: perceptions of pharmacy staff
Kairuz, Therese E., Bellamy, Kim M., Lord, Elisabeth, Ostini, Remo and Emmerton, Lynne M.. 2015. "Health literacy among consumers in community pharmacy: perceptions of pharmacy staff." Health Expectations. 18 (5), pp. 1041-1051. https://doi.org/10.1111/hex.12077
Access to medication and pharmacy services for resettled refugees: a systematic review
Bellamy, Kim, Ostini, Remo, Martini, Nataly and Kairuz, Therese. 2015. "Access to medication and pharmacy services for resettled refugees: a systematic review." Australian Journal of Primary Health. 21 (3), pp. 273-278. https://doi.org/10.1071/PY14121
A Psychometric Evaluation of Three Pain Rating Scales for People with Moderate to Severe Dementia
Neville, Christine and Ostini, Remo. 2014. "A Psychometric Evaluation of Three Pain Rating Scales for People with Moderate to Severe Dementia." Pain Management Nursing. 15 (4), pp. 798-806. https://doi.org/10.1016/j.pmn.2013.08.001
Localised data sources for community health indicators: Ipswich Local Governance Area
Ostini, Jenny, Ostini, Remo and Bush, Robert. 2010. Localised data sources for community health indicators: Ipswich Local Governance Area. Healthy Communities Reseach Centre, The University of Queensland. https://doi.org/10.13140/2.1.4938.3686
Use of Ezetimibe in Australia
Hollingworth, Samantha A., Ostini, Remo, Michael, David, Martin, Jennifer and Susan, Tett. 2014. "Use of Ezetimibe in Australia." 30th International Conference on Pharmacoepidemiology and Therapeutic Risk Management (2014). Taiwan 24 - 27 Oct 2014 United Kingdom. https://doi.org/10.1002/pds.3701
Fit & fuelled kids: program evaluation report 2010
Ostini, Jenny and Ostini, Remo. 2011. Fit & fuelled kids: program evaluation report 2010. Brisbane, Australia. University of Queensland. https://doi.org/10.13140/2.1.3463.8085
Knowledge of human papillomavirus (HPV) testing in the USA, the UK and Australia: an international survey
Dodd, Rachael H., McCaffery, Kirsten J., Marlow, Laura A. V., Ostini, Remo, Zimet, Gregory D. and Waller, Jo. 2014. "Knowledge of human papillomavirus (HPV) testing in the USA, the UK and Australia: an international survey." Sexually Transmitted Infections. 90 (3), pp. 201-207. https://doi.org/10.1136/sextrans-2013-051402
Investigating the association between health literacy and nonadherence
Ostini, Remo and Kairuz, Therese. 2014. "Investigating the association between health literacy and nonadherence." International Journal of Clinical Pharmacy. 36 (1), pp. 36-44. https://doi.org/10.1007/s11096-013-9895-4
The characteristics of inactive men working in a regional area of Queensland, Australia
Wong, Jason Y. L., Gilson, Nicholas D., Peeters, G. M. E. E. (Geeske), Ostini, Remo, Bush, Robert A. and Brown, Wendy J.. 2014. "The characteristics of inactive men working in a regional area of Queensland, Australia." Journal of Science and Medicine in Sport. 17 (1), pp. 56-60. https://doi.org/10.1016/j.jsams.2013.02.008
Knowledge of human papillomavirus (HPV) and HPV vaccination: An international comparison
Marlow, Laura A. V., Zimet, Gregory D., McCaffery, Kirsten J., Ostini, Remo and Waller, Jo. 2013. "Knowledge of human papillomavirus (HPV) and HPV vaccination: An international comparison." Vaccine. 31 (5), pp. 763-769. https://doi.org/10.1016/j.vaccine.2012.11.083
Validation of a measure of knowledge about human papillomavirus (HPV) using item response theory and classical test theory
Waller, Jo, Ostini, Remo, Marlow, Laura A. V., McCaffery, Kirsten and Zimet, Gregory. 2013. "Validation of a measure of knowledge about human papillomavirus (HPV) using item response theory and classical test theory." Preventive Medicine. 56 (1), pp. 35-40. https://doi.org/10.1016/j.ypmed.2012.10.028
Quality Use of Medicines – medication safety issues in naming; look-alike, sound-alike medicine names
Ostini, Remo, Roughead, Elizabeth E., Kirkpatrick, Carl M. J., Monteith, Greg R. and Tett, Susan E.. 2012. "Quality Use of Medicines – medication safety issues in naming; look-alike, sound-alike medicine names." International Journal of Pharmacy Practice. 20 (6), pp. 349-57. https://doi.org/10.1111/j.2042-7174.2012.00210.x
The Audit of Diabetes-Dependent Quality of Life 19 (ADDQoL): feasibility, reliability and validity in a population-based sample of Australian adults
Ostini, Remo, Dower, Jo and Donald, Maria. 2012. "The Audit of Diabetes-Dependent Quality of Life 19 (ADDQoL): feasibility, reliability and validity in a population-based sample of Australian adults." Quality of Life Research. 21 (8), pp. 1471-1477. https://doi.org/10.1007/s11136-011-0043-0